Method and device for determining coronary plaque type, electronic equipment and storage medium

Abstract

The application provides a coronary plaque type determination method and device, electronic equipment and storage medium. The determination method comprises: inputting any one blood vessel section in an acquired unlabeled coronary sample image and corresponding unlabeled multiple intracavity image sections into an image registration training module to obtain a target intracavity image section most matched with each blood vessel section in the coronary sample image; inputting any one blood vessel section in the coronary sample image and the target intracavity image section corresponding to the blood vessel section into a plaque recognition training module to train the plaque recognition training module; determining the trained plaque recognition training module as a coronary plaque recognition model, and inputting a coronary image into the coronary plaque recognition model to obtain a coronary plaque type result. The technical solution provided by the application can improve the accuracy of CTA in recognizing the coronary plaque type.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing, and in particular to a coronary plaque type determination method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Coronary plaque (hereinafter referred to as "coronary plaque") is a hardening plaque inside the coronary artery, mostly caused by increased blood lipids, blood impurities accumulation, and elevated blood sugar, etc. Coronary plaque can cause secondary hypertension, affect local blood supply of the human body, and also increase the burden on the heart, threatening the health of patients. Coronary plaque has multiple types, so it is necessary to determine the specific type of coronary plaque of a patient, which can specify a targeted treatment plan for the patient.

[0003] At present, there are two ways to identify the type of coronary plaque. One is through invasive examination of intracoronary images, such as intravascular ultrasound (IVUS) and optical coherence tomography (OCT), which has high identification accuracy, but is expensive and has certain risks, and is not suitable for routine examination. The other is non-invasive examination of CT angiography (CTA), which has the characteristics of wide applicability, low price, non-invasiveness, and simple operation, and is widely used in various disease diagnoses. However, CTA can only identify three types of plaque: calcified plaque, non-calcified plaque, and mixed plaque, resulting in low resolution of CTA in identifying coronary plaque. Therefore, how to improve the accuracy of CTA in identifying the type of coronary plaque has become a problem to be solved. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a coronary plaque type determination method and device, an electronic device, and a storage medium, which can match unlabeled coronary artery sample images (CTA) and intracavity sample images (such as OCT), use large-scale unlabeled data to improve the generalization ability and effect of the model, and directly use the matching results for plaque identification in the next stage. In the plaque identification training stage, the high-resolution feature of the intracavity sample image (which can identify more and finer types of plaque) is used to guide the model to identify more features in the coronary artery sample image, thereby improving the accuracy of CTA in identifying the type of coronary plaque.

[0005] The present application mainly includes the following aspects:

[0006] In a first aspect, the embodiments of the present application provide a coronary plaque type determination method, which comprises:

[0007] The determination method comprises:

[0008] Obtaining a coronary artery image;

[0009] Inputting the coronary artery image into a coronary plaque recognition model to output a type result of a coronary plaque corresponding to the coronary artery image;

[0010] The coronary plaque recognition model is obtained by the following steps:

[0011] Obtaining a coronary artery sample image and a corresponding intraluminal sample image from a pre-created sample data set; wherein the coronary artery sample image and the intraluminal sample image are both unlabeled images; the coronary artery sample image includes multiple vessel sections along a vessel centerline; and the intraluminal sample image includes multiple intraluminal image sections along a shooting direction;

[0012] Inputting any one of the vessel sections in the coronary artery sample image and the multiple intraluminal image sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, to obtain a trained image registration training module;

[0013] Obtaining, by the trained image registration training module, a target intraluminal image section that is most matched with each of the vessel sections in the coronary artery sample image from the multiple intraluminal image sections corresponding to the coronary artery sample image;

[0014] Inputting any one of the vessel sections in the coronary artery sample image and the target intraluminal image section corresponding to the vessel section into a plaque recognition training module to train the plaque recognition training module, to obtain a trained plaque recognition training module;

[0015] Determining the trained plaque recognition training module as a coronary plaque recognition model.

[0016] Further, the step of inputting any one of the vessel sections in the coronary artery sample image and the multiple intraluminal image sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, to obtain a trained image registration training module, comprises:

[0017] Inputting any one of the vessel sections in the coronary artery sample image into a first feature extraction layer of the image registration training module, to obtain a coronary artery feature of the vessel section;

[0018] inputting the plurality of intraluminal image sections in the intraluminal sample image corresponding to the coronary sample image into a second feature extraction layer of the image registration training module to obtain intraluminal image features of each intraluminal image section;

[0019] determining a target intraluminal image section most matched with the vascular section from the plurality of intraluminal image sections based on the coronary artery features of the vascular section and the intraluminal image features of each intraluminal image section;

[0020] determining a label of each intraluminal image section corresponding to the vascular section based on the target intraluminal image section;

[0021] obtaining a first loss function based on the label of each intraluminal image section;

[0022] determining whether the first loss function converges;

[0023] if not, updating parameters of the image registration training module, and obtaining a next coronary sample image and an intraluminal sample image corresponding to the next coronary sample image to continue training the image registration training module until the first loss function converges;

[0024] if yes, obtaining the trained image registration training module.

[0025] Further, the step of inputting any one of the vascular sections in the coronary sample image and a target intraluminal image section corresponding to the vascular section into the plaque identification training module to train the plaque identification training module to obtain the trained plaque identification training module, comprises:

[0026] inputting any one of the vascular sections in the coronary sample image and a target intraluminal image section corresponding to the vascular section into the plaque identification training module, extracting coronary artery features in the vascular section and intraluminal image features of the target intraluminal image section corresponding to the vascular section;

[0027] obtaining a current influence coefficient, and fusing the coronary artery features in the vascular section and the intraluminal image features of the target intraluminal image section corresponding to the vascular section through the current influence coefficient to obtain fusion features;

[0028] after the fusion features pass through a full connection layer and normalization processing, obtaining probabilities that a coronary plaque represented by the fusion features belongs to each preset classification;

[0029] obtaining a second loss function based on the probabilities that the coronary plaque represented by the fusion features belongs to each preset classification;

[0030] determining whether the second loss function converges;

[0031] If no, the parameters of the plaque recognition training module and the current influence coefficient are updated, the next coronary artery sample image and the target intraluminal sample image corresponding to the next coronary artery sample image are obtained, and the plaque recognition training module is continuously trained until the second loss function converges.

[0032] If yes, the trained plaque recognition training module is obtained.

[0033] Further, the step of inputting the coronary artery image into the coronary plaque recognition model and outputting the type result of the coronary plaque corresponding to the coronary artery image comprises:

[0034] For each blood vessel section in the coronary artery image, the blood vessel section in the coronary artery image is input into the coronary plaque recognition model for feature extraction to obtain the coronary artery feature of the blood vessel section in the coronary artery image.

[0035] After the coronary artery feature of the blood vessel section in the coronary artery image is processed by a full connection layer and normalized, the probability that the coronary plaque represented by the coronary artery feature of the blood vessel section in the coronary artery image belongs to each preset classification is obtained.

[0036] In the probability, the preset classification corresponding to the probability with the largest value is determined as the type result of the coronary plaque corresponding to the blood vessel section in the coronary artery image.

[0037] Further, the sample data set is created by the following steps:

[0038] Obtain multiple initial images of the patient for non-invasive coronary artery scanning and intraluminal sample images of the patient for invasive intraluminal scanning;

[0039] The multiple initial images of the patient are processed to obtain the coronary artery sample image of the patient.

[0040] For each patient, the coronary artery sample image of the patient and the intraluminal sample image of the patient are stored as a data pair of the patient.

[0041] After storing the data pair of each patient, the sample data set is obtained.

[0042] Further, the step of processing the multiple initial images of the patient to obtain the coronary artery sample image of the patient comprises:

[0043] The multiple initial images of the patient are preprocessed to obtain a three-dimensional blood vessel image of the patient.

[0044] Segment the three-dimensional blood vessel image to obtain a coronary vessel image;

[0045] Extract a blood vessel centerline in the coronary vessel image, and determine a blood vessel cross section perpendicular to the blood vessel centerline along the blood vessel centerline;

[0046] Obtain an image interpolation result of each blood vessel cross section, and splice the image interpolation result of each blood vessel cross section to obtain a coronary artery sample image of the patient.

[0047] Further, the plaque recognition training module comprises a third feature extraction layer and a fourth feature extraction layer; the third feature extraction layer is configured to extract a coronary artery feature in the blood vessel cross section; the fourth feature extraction layer is configured to extract an intraluminal image feature of the target intraluminal image cross section; parameters of the plaque recognition training module comprise parameters of the third feature extraction layer and parameters of the fourth feature extraction layer; the third feature extraction layer and the fourth feature extraction layer are obtained by the following steps:

[0048] The parameters in the first feature extraction layer in the trained image registration training module are used as parameters of a network for extracting a coronary artery feature in the blood vessel cross section in the plaque recognition training module, and the network with the parameters for extracting the coronary artery feature in the blood vessel cross section is determined as the third feature extraction layer.

[0049] The parameters in the second feature extraction layer in the trained image registration training module are used as parameters of a network for extracting an intraluminal image feature of the target intraluminal image cross section in the plaque recognition training module, and the network with the parameters for extracting the intraluminal image feature of the target intraluminal image cross section is determined as the fourth feature extraction layer.

[0050] In a second aspect, the embodiments of the present application further provide a determination device of a coronary plaque type, the determination device comprising:

[0051] An acquisition module configured to acquire a coronary artery image;

[0052] A determination module configured to input the coronary artery image into a coronary plaque recognition model, and output a type result of a coronary plaque corresponding to the coronary artery image;

[0053] A training module configured to train the coronary plaque recognition model; the training module comprises an acquisition unit, a first training unit, a matching unit, a second training unit, and a determination unit.

[0054] The acquisition unit is configured to acquire a coronary artery sample image and a lumen sample image corresponding to the coronary artery sample image from a pre-created sample data set; the coronary artery sample image and the lumen sample image are both unlabeled images; the coronary artery sample image includes a plurality of blood vessel sections along a blood vessel center line; and the lumen sample image includes a plurality of lumen image sections along a shooting direction.

[0055] The first training unit is configured to input any one blood vessel section in the coronary artery sample image and a plurality of lumen image sections in the lumen sample image corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, so as to obtain a trained image registration training module.

[0056] The matching unit is configured to obtain, by using the trained image registration training module, a target lumen image section most matched with each blood vessel section in the coronary artery sample image from the plurality of lumen image sections corresponding to the coronary artery sample image.

[0057] The second training unit is configured to input any one blood vessel section in the coronary artery sample image and a target lumen image section corresponding to the blood vessel section into a plaque recognition training module to train the plaque recognition training module, so as to obtain a trained plaque recognition training module.

[0058] The determination unit is configured to determine the trained plaque recognition training module as a coronary plaque recognition model.

[0059] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the coronary plaque type determination method as described above.

[0060] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the coronary plaque type determination method as described above.

[0061] The method for determining the type of coronary plaque provided by the embodiment of the application, the device, the electronic equipment and the storage medium, the method comprises: acquiring a coronary artery image; inputting the coronary artery image into a coronary plaque recognition model to output a type result of a coronary plaque corresponding to the coronary artery image; wherein the coronary plaque recognition model is obtained by training through the following steps: acquiring a coronary artery sample image and a intraluminal sample image corresponding to the coronary artery sample image from a pre-created sample data set; wherein the coronary artery sample image and the intraluminal sample image are both unlabeled images; the coronary artery sample image comprises a plurality of blood vessel sections along a blood vessel center line; the intraluminal sample image comprises a plurality of intraluminal image sections along a shooting direction; inputting any one of the blood vessel sections in the coronary artery sample image and a plurality of intraluminal image sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module to obtain a trained image registration training module; obtaining a target intraluminal image section most matched with each blood vessel section in the coronary artery sample image from the plurality of intraluminal image sections corresponding to the coronary artery sample image through the trained image registration training module; inputting any one of the blood vessel sections in the coronary artery sample image and the target intraluminal image section corresponding to the blood vessel section into a plaque recognition training module to train the plaque recognition training module to obtain a trained plaque recognition training module; and determining the trained plaque recognition training module as the coronary plaque recognition model.

[0062] In this way, by using the technical solution provided by the application, the unlabeled coronary artery sample image (CTA) and the intraluminal sample image (for example, OCT) are matched, the generalization ability and effect of the model are improved by using large-scale unlabeled data, and the matching result is directly used for plaque recognition in the next stage. In the plaque recognition training stage, the characteristics of the intraluminal sample image with high resolution (which can identify more and finer types of plaque types) are used to guide the model to identify more features in the coronary artery sample image, thereby improving the accuracy of CTA in identifying the type of coronary plaque.

[0063] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0065] Figure 1 A flow chart of a method for determining a coronary plaque type is shown;

[0066] Figure 2 A flow chart of another method for determining a coronary plaque type is shown;

[0067] Figure 3 A schematic diagram of a workflow of an image registration module is shown;

[0068] Figure 4 A schematic diagram of a workflow of a plaque recognition training module is shown;

[0069] Figure 5 A structure diagram of a device for determining a coronary plaque type is shown;

[0070] Figure 6 A structure diagram of a device for determining a coronary plaque type is shown;

[0071] Figure 7 A structure diagram of an electronic device is shown. DETAILED DESCRIPTION

[0072] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts by those skilled in the art under the guidance of the content of the present application.

[0073] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0074] In order to enable those skilled in the art to use the content of the present application, the following embodiments are given in combination with a specific application scenario "determination of coronary plaque type", and the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of the present application.

[0075] The method, device, electronic device or computer readable storage medium described in the embodiments of the present application can be applied to any scenario requiring determination of coronary plaque type, and the embodiments of the present application do not limit the specific application scenario. Any scheme using the method, device, electronic device and storage medium provided by the embodiments of the present application for determining coronary plaque type is within the protection scope of the present application.

[0076] It is worth noting that coronary plaque (hereinafter referred to as "coronary plaque") is the existence of hardening plaque inside the coronary artery, which is mostly caused by increased blood lipids, blood impurities accumulation and high blood sugar, etc. Coronary plaque can cause secondary hypertension, affect local blood supply of the human body, and also increase the burden on the heart, threatening the health of the patient. Coronary plaque has multiple types, so it is necessary to determine the specific type of coronary plaque of the patient, so as to specify a targeted treatment plan for the patient.

[0077] At present, there are two ways to identify the type of coronary plaque. One is through invasive examination of intracoronary image, such as intravascular ultrasound (IVUS) and optical coherence tomography (OCT), which has high identification accuracy, but is high in cost and has certain risks, and is not suitable for routine examination. The other is non-invasive examination of CT angiography (CTA), which has the characteristics of wide applicability, low price, non-invasive and simple operation, and is widely used in various disease diagnoses. However, CTA can only identify calcified plaque, non-calcified plaque and mixed plaque, which leads to low resolution of CTA in identifying coronary plaque. Therefore, how to improve the accuracy of CTA in identifying the type of coronary plaque has become a problem to be solved.

[0078] Based on this, the application provides a coronary plaque type determination method and device, electronic equipment and storage medium. The determination method comprises: acquiring a coronary artery image; inputting the coronary artery image into a coronary plaque recognition model to output a type result of a coronary plaque corresponding to the coronary artery image; wherein the coronary plaque recognition model is obtained by training the following steps: acquiring a coronary artery sample image and a intraluminal sample image corresponding to the coronary artery sample image from a pre-created sample data set; wherein the coronary artery sample image and the intraluminal sample image are both unlabeled images; the coronary artery sample image comprises multiple vessel sections along a vessel center line; the intraluminal sample image comprises multiple intraluminal image sections along a shooting direction; inputting any one vessel section in the coronary artery sample image and multiple intraluminal image sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module to obtain a trained image registration training module; obtaining a target intraluminal image section most matched with each vessel section in the coronary artery sample image from the multiple intraluminal image sections corresponding to the coronary artery sample image through the trained image registration training module; inputting any one vessel section in the coronary artery sample image and the target intraluminal image section corresponding to the vessel section into a plaque recognition training module to train the plaque recognition training module to obtain a trained plaque recognition training module; and determining the trained plaque recognition training module as the coronary plaque recognition model.

[0079] In this way, the technical scheme provided by the application can match unlabeled coronary artery sample images (CTA) and intraluminal sample images (such as OCT), use large-scale unlabeled data to improve the generalization ability and effect of the model, and directly use the matching result for plaque recognition in the next stage. In the plaque recognition training stage, the high-resolution feature of the intraluminal sample image (which can identify more and finer types of plaque types) is used to guide the model to identify more features in the coronary artery sample image, thereby improving the accuracy of CTA in identifying coronary plaque types.

[0080] To make the application more comprehensible, the technical scheme provided by the application will be described in detail below with reference to specific embodiments.

[0081] Please refer to Figure 1 , Figure 1 A flowchart of a coronary plaque type determination method provided by an embodiment of the application is shown in Figure 1 , and the determination method comprises:

[0082] S101, acquiring a coronary artery image;

[0083] In this step, the coronary artery image is a coronary CTA, which is a medical imaging technique that uses computed tomography (CT) to generate three-dimensional images of the coronary arteries. This technique can show the anatomical structure of the coronary arteries, stenosis or obstruction, and is used to evaluate coronary heart disease, heart valve disease and other cardiovascular diseases. Coronary CTA usually involves intravenous injection of contrast agent, followed by image reconstruction using a CT scanner, which can provide high-resolution vascular imaging. Generally, CTA can only identify three types of plaque: calcified plaque, non-calcified plaque and mixed plaque.

[0084] S102, input the coronary artery image into the coronary plaque recognition model, and output the type result of the coronary plaque corresponding to the coronary artery image;

[0085] In this step, the coronary plaque recognition model includes an image registration training module and a plaque recognition training module in the training process, and includes only the plaque recognition training module in the application process.

[0086] It should be noted that the coronary plaque recognition model needs to be obtained before step S102. Please refer to Figure 2 , Figure 2 The flowchart of another coronary plaque type determination method provided by the embodiment of the present application is shown in Figure 2 The coronary plaque recognition model is trained by the following steps:

[0087] S201, obtaining a coronary artery sample image and a lumen sample image corresponding to the coronary artery sample image from a pre-created sample data set;

[0088] In this step, the coronary artery sample image and the lumen sample image are both unlabeled images; the coronary artery sample image includes multiple vessel cross sections along the vessel centerline; and the lumen sample image includes multiple lumen image cross sections along the shooting direction.

[0089] Here, the intraluminal sample image is a coronary intraluminal image, mainly divided into IVUS and OCT. IVUS sends a miniature ultrasound probe into the blood vessel lumen through catheter technology, performs 360-degree scanning in the blood vessel, and clearly displays the heart blood vessel structure and lesions on the display screen. Unlike coronary angiography, which visualizes the coronary artery by filling the lumen with contrast medium, IVUS displays cross-sectional images of the blood vessel, providing intraluminal images of the in-vivo blood vessel. IVUS can accurately measure the lumen and vessel diameter and determine the severity and nature of the lesion, playing a very important role in improving the understanding of coronary artery lesions and guiding interventional treatment. OCT is a high-resolution cardiovascular imaging technology. Its principle is to place an imaging catheter inside the blood vessel, analyze the time delay of the light source reflected to the tube wall tissue, and convert the internal structure information into high-resolution images displayed on the display. OCT uses optical coherence tomography to generate high-definition cardiovascular structure images, which can provide high-resolution images for assessing coronary intimal thickness, plaque characteristics, and plaque stability. It has high sensitivity and specificity for diagnosing and evaluating cardiovascular diseases such as coronary heart disease and atherosclerosis. OCT generally uses an optical probe guided by a guide wire to enter the coronary artery, and then uses optical coherence tomography to generate high-resolution images. Intraluminal images can show calcified plaques, lipid plaques, fibrous plaques, mixed plaques, and thrombi.

[0090] It should be noted that the sample data set is created by the following steps:

[0091] 1) Obtain multiple initial images of a patient undergoing non-invasive coronary scanning and intraluminal sample images of the patient undergoing invasive intraluminal scanning;

[0092] In this step, the patient undergoes coronary CTA scanning, which is a non-invasive examination method performed by a computed tomography scanner (CT). During the scanning process, the patient lies on the scanning bed, and the CT machine rotates around the body to generate a series of X-ray images, i.e., multiple initial images. The patient's coronary sample images are obtained from the multiple initial images, and the corresponding intraluminal sample images are found based on the patient and coronary sample images collected in the CTA. For example, there is a blood vessel LAD of patient A in the CTA data, and the blood vessel has also been examined by intraluminal imaging. Therefore, an effective data pair can be formed.

[0093] Here, generally, the patient first undergoes CTA examination, which is less expensive and less harmful. When CTA confirms the presence of lesions and further observation of the lesions is required, intraluminal imaging is performed, which is more expensive and requires the patient to be on the operating table for detection, causing some harm to the human body. When creating a sample data set, patient data that has undergone both CTA examination and intraluminal imaging can be obtained.

[0094] 2) data processing of the multiple initial images of the patient to obtain a coronary artery sample image of the patient;

[0095] It should be noted that the step of data processing of the multiple initial images of the patient to obtain a coronary artery sample image of the patient comprises:

[0096] (1) preprocessing of the multiple initial images of the patient to obtain a three-dimensional blood vessel image of the patient;

[0097] In this step, the preprocessing is the process of generating a three-dimensional blood vessel image by the computer from the initial images.

[0098] (2) segmentation processing of the three-dimensional blood vessel image to obtain a coronary artery vessel image;

[0099] In this step, in the three-dimensional blood vessel image, the coronary artery vessel image can be extracted by threshold segmentation and other methods.

[0100] (3) extraction of a blood vessel center line in the coronary artery vessel image, and determination of a blood vessel cross section perpendicular to the blood vessel center line along the blood vessel center line;

[0101] In this step, based on the segmentation result of step (2) above, the center line of the blood vessel is obtained by a minimum loss distance algorithm, and after obtaining the blood vessel center line, some post-processing can be performed, such as branch removal, smoothing, and gray scale transformation, to obtain a better visualization effect; then along the center line, a cross section perpendicular to the center line (i.e. a blood vessel cross section) is obtained.

[0102] (4) obtaining an image interpolation result of each blood vessel cross section, and splicing the image interpolation result of each blood vessel cross section to obtain a coronary artery sample image of the patient.

[0103] In this step, the image interpolation result of each cross section obtained in step (3) above is obtained, and these cross sections are spliced to obtain a final blood vessel straightening image (i.e. a coronary artery sample image). Here, since the intraluminal sample image itself is a straightening image along the shooting direction, it is not necessary to perform straightening processing again.

[0104] 3) for each patient, storing the coronary artery sample image of the patient and the intraluminal sample image of the patient as a data pair of the patient;

[0105] 4) after storing the data pair of each patient, obtaining a sample data set.

[0106] S202, inputting any one blood vessel section in the coronary artery sample image and multiple intraluminal image sections in the intraluminal sample image corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, to obtain a trained image registration training module;

[0107] In this step, an image registration module needs to be constructed. First, an intraluminal image feature extraction network (i.e., a second feature extraction layer) is constructed. Since the resolution of the intraluminal image is high (commonly 512x512), a deep network such as ResNet is used, and the specific network is not limited. The coronary artery feature extraction network (i.e., a first feature extraction layer) can use a relatively shallow neural network such as VGG since the CTA section is small (commonly 64x64). Here, the number of output layer features of the two is unified, such as setting a feature vector with a length of 1024.

[0108] It should be noted that the step of inputting any one blood vessel section in the coronary artery sample image and multiple intraluminal image sections in the intraluminal sample image corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, to obtain a trained image registration training module, comprises:

[0109] S2021, inputting any one blood vessel section in the coronary artery sample image into a first feature extraction layer of the image registration training module, to obtain coronary artery features of the blood vessel section;

[0110] In this step, the input of the first feature extraction layer is a single CTA section (blood vessel section), and the input of the second feature extraction layer is multiple intraluminal image sections. These sections are randomly selected from the same patient's unified blood vessels. The single CTA section is selected in the Patch CTA mode:

[0111] Patch CTA = f CTA (n) | n = random (N, 1);

[0112] Where f CTA (n) represents selecting the nth section (blood vessel section) from the CTA blood vessel straight image (coronary artery sample image), and random (N, 1) represents randomly selecting an integer from 1 to N, and N is the total number of CTA blood vessel straight image sections.

[0113] S2022, inputting multiple intraluminal image sections in the intraluminal sample image corresponding to the coronary artery sample image into a second feature extraction layer of the image registration training module, to obtain intraluminal image features of each intraluminal image section;

[0114] In this step, the selection of the multiple intraluminal image sections is as follows:

[0115] Patch Intra = f Intra ({m}) | {m} = random(M, m);

[0116] Wherein, {m} is a set, indicating that there are m integers, and random(M, m) indicates that m integers are randomly selected from 1 to M, and M is the total number of intraluminal image sections in the intraluminal sample image.

[0117] S2023, based on the coronary artery features of the vascular section and the intraluminal image features of each intraluminal image section, determining a target intraluminal image section most matched with the vascular section among the multiple intraluminal image sections;

[0118] For example, the output of the first feature extraction layer is a feature vector F CTA ∈R 1024×1 , and the output of the second feature extraction layer is a feature vector F Intra ∈R m×1024 Then, the cosine similarity r(F CTA , F Intra ) between them is calculated:

[0119]

[0120] Wherein, F Intra represents the intraluminal image features of the intraluminal image section, F CTA represents the coronary artery features of the vascular section, F Intra × F CTA represents the matrix multiplication of the intraluminal image features of the intraluminal image section and the coronary artery features of the vascular section, and the result is an m×1 vector, i.e. the similarity of the vascular section with the m intraluminal image sections, and the intraluminal image section with the largest similarity is determined as the target intraluminal image section most matched with the vascular section.

[0121] S2024, based on the target intraluminal image section, determining the label of each intraluminal image section corresponding to the vascular section;

[0122] In this step, based on the similarity calculation result of step S2023, the position (target intraluminal image section) with the largest one of the similarities is selected, and it is set as label 1, and the remaining items (other intraluminal image sections except the target intraluminal image section) are labeled as 0, and the specific formula is as follows:

[0123]

[0124] x = argmax(r(FCTA , F Intra ));

[0125] wherein, l(i) is the label of the i-th endoluminal image section, and x is the position of the target endoluminal image section in the endoluminal sample image.

[0126] S2025, obtaining a first loss function based on the label of each endoluminal image section;

[0127] In this step, after the similarity result and the automatic label are obtained, the first loss function Loss1 function can be calculated. Here, a commonly used binary classification loss function CrossEntropy can be used:

[0128]

[0129] wherein, p(i) represents the predicted value of the i-th element in the result vector, i.e., the similarity between the blood vessel section and the i-th endoluminal image section.

[0130] S2026, determining whether the first loss function converges;

[0131] S2027, if not, updating the parameters of the image registration training module, and obtaining a next coronary sample image and an endoluminal sample image corresponding to the next coronary sample image to continue training the image registration training module until the first loss function converges.

[0132] S2028, if yes, obtaining the trained image registration training module.

[0133] In the above steps S2026 to S2028, the randomly sampled data pairs (coronary sample image and endoluminal sample image corresponding to the coronary sample image) are continuously input into the image registration training module, the output is calculated Loss1, and then the gradient descent method is used to make Loss1 continuously decrease in the training iteration process, so as to eventually not decrease (i.e., converge), thereby completing the training of the image registration training module.

[0134] S203, obtaining the target endoluminal image section most matched with each blood vessel section in the coronary sample image from a plurality of endoluminal image sections corresponding to the coronary sample image by using the trained image registration training module.

[0135] In this step, the step of obtaining the target endoluminal image section is described in detail in the step S203. Figure 3 , Figure 3 is a schematic diagram of an image registration module workflow provided by an embodiment of the present application, as shown in Figure 3As shown in the middle, the CTA cross section (i.e. the blood vessel cross section) and the corresponding multiple intraluminal image cross sections are obtained, the CTA cross section is input into the first feature extraction layer to extract the CTA features (i.e. the coronary artery features) in the CTA cross section, the multiple intraluminal image cross sections are input into the second feature extraction layer to extract the intraluminal image features in each intraluminal image cross section, the intraluminal image features in each intraluminal image cross section and the CTA features are compared for similarity, and the similarity of the CTA features and the intraluminal image features in each intraluminal image cross section is obtained, for example, the similarity of the CTA features and the intraluminal image features in the first intraluminal image cross section is 0.9, the similarity of the CTA features and the intraluminal image features in the second intraluminal image cross section is 0.7, and so on. Through the similarity of the CTA features and the intraluminal image features in each intraluminal image cross section, the corresponding label of each intraluminal image cross section can be obtained, that is, the label of the intraluminal image cross section corresponding to the maximum value is set to 1, and the labels of the other intraluminal image cross sections are set to 0, indicating that the intraluminal image cross section with the label of 1 is the target intraluminal image cross section that best matches the CTA cross section.

[0136] Here, after obtaining the trained image registration module, the data for training the plaque recognition training module also needs to be prepared before training the plaque recognition training module. First, all CTA cross sections are sequentially input into the trained image registration module. For each CTA cross section, a maximum similarity intraluminal image cross section can be found as a target intraluminal image cross section. Each CTA cross section and the corresponding target intraluminal image cross section are saved. Repeat this process until all CTA cross sections have found the corresponding target intraluminal image cross section. Then only the matched target intraluminal image cross section needs to be labeled (the number is small). For example, the classification labels that can be labeled include no plaque, calcified plaque, lipid plaque, fibrous plaque, mixed plaque, and thrombus, a total of 6 types (classification labels are marked as 0-5, respectively). For example, if a target intraluminal image cross section contains a calcified plaque, the target intraluminal image cross section is marked as 1.

[0137] S204, inputting any one blood vessel cross section in the coronary artery sample image and the target intraluminal image cross section corresponding to the blood vessel cross section into the plaque recognition training module to train the plaque recognition training module, and obtaining a trained plaque recognition training module;

[0138] In this step, the network architecture of the plaque recognition training module is consistent with that of the image registration module, and a fusion feature layer is added.

[0139] It should be noted that the step of inputting any one blood vessel cross section in the coronary artery sample image and the target intraluminal image cross section corresponding to the blood vessel cross section into the plaque recognition training module to train the plaque recognition training module, and obtaining a trained plaque recognition training module, includes:

[0140] S2041. Input any blood vessel cross section and the target intraluminal image cross section corresponding to the blood vessel cross section into the plaque recognition training module, and extract the coronary artery features in the blood vessel cross section and the intraluminal image features of the target intraluminal image cross section corresponding to the blood vessel cross section;

[0141] It should be noted that the plaque recognition training module includes a third feature extraction layer and a fourth feature extraction layer; the third feature extraction layer is used to extract coronary artery features from the vascular cross-section; the fourth feature extraction layer is used to extract intraluminal image features from the target intraluminal image cross-section; the third and fourth feature extraction layers are obtained through the following steps:

[0142] 1. The parameters in the first feature extraction layer of the trained image registration training module are used as the parameters of the network used to extract coronary artery features in the vascular cross section in the plaque recognition training module, and the network with parameters used to extract coronary artery features in the vascular cross section is determined as the third feature extraction layer;

[0143] Second, the parameters in the second feature extraction layer of the trained image registration training module are used as the parameters of the network in the patch recognition training module used to extract intracavitary image features of the target intracavitary image cross section, and the network with parameters used to extract intracavitary image features of the target intracavitary image cross section is determined as the fourth feature extraction layer.

[0144] In steps one and two above, the training of the patch recognition module, since the intracavitary image feature extraction network and CTA feature extraction network have the same structure as the image registration module, can fully utilize the advantages of the large-scale training in the previous step. The parameters of the extraction networks (first and second feature extraction layers) of the image registration module trained in the previous steps can be used as the initialization parameters of the current network (third and fourth feature extraction layers). This approach allows the model to converge quickly, achieving better training results. Therefore, during the first training, the initial parameters of the third feature extraction layer can use the parameters of the first feature extraction layer, and the initial parameters of the fourth feature extraction layer can use the parameters of the second feature extraction layer. Then, during subsequent training, when updating the parameters of the patch recognition training module, this includes updating the parameters of the third and fourth feature extraction layers to train them.

[0145] S2042. Obtain the current influence coefficient. Using the current influence coefficient, fuse the coronary artery features in the cross-section of the blood vessel with the intraluminal image features of the target intraluminal image cross-section corresponding to the cross-section of the blood vessel to obtain the fused features.

[0146] In this step, the plaque recognition training module further comprises a fusion feature layer, and the current influence coefficient, the coronary artery features in the current blood vessel cross section, and the intraluminal image features of the target intraluminal image cross section corresponding to the current blood vessel cross section are input into the fusion feature layer to obtain fusion features, which can be specifically represented as:

[0147] F fuse = F Intra × τ + F CTA ;

[0148] wherein, F fuse ∈ R 1024×1 represents the fused features (i.e., fusion features), τ represents the current influence coefficient, which is a dynamically changing value and can be initially set to 1.

[0149] Here, τ will continuously decrease during the training process until it is 0, and the current influence coefficient can be determined by the current training round of the plaque recognition training module, the step length of τ reduction each time, and the initial value of τ set in advance. For example, the current training round of the plaque recognition training module is the epoch round, and it is reduced by 0.01 (step length) each round until it is 0. The specific formula is as follows:

[0150] τ epoch = 1-0.01×epoch;

[0151] wherein, τ epoch represents the value of the current influence coefficient when the training round is the epoch round; 1 is the initial value of τ set in advance, and 0.01 is the step length of τ reduction each time set in advance.

[0152] S2043, after the fusion features are processed through a full connection layer and normalization, the probability that the coronary plaque represented by the fusion features belongs to each preset classification is obtained;

[0153] In this step, after the features are fused, a final result is obtained through a full connection layer and a normalized softmax layer:

[0154] F out = softmax(Linear(F fuse ));

[0155] wherein, Linear is a full connection layer, and if there are 6 preset classifications, the parameter dimension of Linear is R 6×1024 , and therefore the result dimension of Linear(F fuse ) is R 6×1 , corresponding to the probability F out of each preset classification.

[0156] S2044, obtaining a second loss function based on a probability that the coronary plaque represented by the fusion feature belongs to each preset classification;

[0157] In this step, the calculation of the second loss function Loss2 is performed. Here, a commonly used multi-classification loss function Multi-CrossEntropy can be used. If there are 6 preset classifications, Loss2 is represented as follows:

[0158]

[0159] Wherein, y(i) represents the result of the i-th channel of the classification label of the current sample (target intraluminal image section), such as the classification label (preset classification) marked on the target intraluminal image section in the previous step, which includes 6 types of no plaque, calcified plaque, lipid plaque, fibrous plaque, mixed plaque and thrombus (the classification labels are marked as 0-5 respectively). If the current sample (target intraluminal image section) is a calcified plaque, y={0, 1, 0, 0, 0, 0}, y(1)=1, y(0)=y(2)=y(3)=y(4)=y(5)=0. p(i) represents the prediction channel result (i.e. the probability that the plaque in the predicted blood vessel section belongs to the i-th preset classification) of the coronary plaque represented by the fusion feature predicted by the plaque recognition training module.

[0160] S2045, determining whether the second loss function converges;

[0161] S2046, if not, updating the parameters of the plaque recognition training module and the current influence coefficient, and obtaining a next coronary artery sample image and a target intraluminal sample image corresponding to the next coronary artery sample image to continue training the plaque recognition training module until the second loss function converges;

[0162] S2047, if yes, obtaining the trained plaque recognition training module.

[0163] In the above steps S2045 to S2047, the plaque recognition training module needs to constantly randomly sample the matched data pairs, input them into the third feature extraction layer and the fourth feature extraction layer respectively, fuse the output features, calculate the second loss function, and then use the gradient descent method to make the second loss function continuously decrease in the training iteration process, so as to eventually not decrease, i.e. the training is completed. Here, the parameters of the plaque recognition training module include the parameters of the third feature extraction layer, the parameters of the fourth feature extraction layer, the parameters of the full connection layer and the parameters of the fusion feature layer, etc. When updating the parameters of the plaque recognition training module in each training, all the parameters included in the plaque recognition training module need to be updated.

[0164] S205, determine the trained plaque recognition training module as a coronary plaque recognition model.

[0165] In this step, in the actual application process, the coronary plaque recognition model is a model for classifying coronary plaques by CTA, so it does not need intraluminal images, only needs to input the CTA section into the coronary plaque recognition model to obtain the type of coronary plaque in the CTA section; therefore, in the trained plaque recognition training module, the plaque recognition module after removing the fourth feature extraction layer and the fusion feature layer is determined as the coronary plaque recognition model.

[0166] For example, please refer to Figure 4 , Figure 4 The working process of the plaque recognition training module provided by the embodiment of the present application is shown in FIG. Figure 4 The CTA section is input into the third feature extraction layer to obtain the CTA feature, the target intraluminal image section is input into the fourth feature extraction layer to obtain the intraluminal image feature, the product of the intraluminal image feature and the current influence coefficient τ is determined as the first feature, the sum of the first feature and the CTA feature is determined as the fusion feature, and the fusion feature is processed by the full connection layer and the normalization processing to obtain the prediction result predicted by the plaque recognition training module, i.e., the probability that the predicted coronary plaque in the CTA section belongs to each preset classification, for example, the probabilities that the predicted coronary plaque in the CTA section belongs to each preset classification are 0.5, 0.1, 0.2, 0.1 and 0 respectively, the label of the preset classification corresponding to the largest probability 0.5 is set to 1, and the labels of the remaining preset classifications are set to 0, and the preset classification with the label 1 is determined as the classification result of the coronary plaque in the CTA section predicted by the plaque recognition training module.

[0167] In step S102, the coronary plaque recognition model is input into the coronary plaque recognition model, and the type result of the coronary plaque corresponding to the coronary artery image is output.

[0168] S1021, for each blood vessel section in the coronary artery image, input the blood vessel section into the coronary plaque recognition model for feature extraction to obtain the coronary artery feature of the blood vessel section in the coronary artery image.

[0169] S1022, after the coronary artery feature of the blood vessel section in the coronary artery image is processed by the full connection layer and the normalization processing, the probability that the coronary plaque represented by the coronary artery feature of the blood vessel section in the coronary artery image belongs to each preset classification is obtained.

[0170] S1023, in the probability, the preset classification corresponding to the largest probability is determined as the type result of the coronary plaque corresponding to the blood vessel section in the coronary artery image.

[0171] In steps S1021 to S1023, when the plaque recognition training module is trained, the current influence coefficient τ is reduced to 0, and the training process is ended. Figure 4 It can also be seen from the above that the fourth feature extraction layer and the fusion feature layer do not contribute, so in the actual application process, the patient only needs to be shot by CTA, without matching the intraluminal image, more rich plaque types can be predicted. The purpose of setting the influence coefficient is to use the intraluminal image features to guide the CTA feature learning to hide the plaque features inside the data (which cannot be seen by the naked eye) in the early training stage, and in the training process, the influence of the intraluminal image network (the fourth feature extraction layer) on the CTA network (the third feature extraction layer) gradually decreases, so that it can finally independently and autonomously classify the plaque. Therefore, in the application, only the CTA section is input into the CTA feature extraction network (the third feature extraction layer), and then the output CTA feature is directly input into the full connection layer Linear layer, and then the final result is obtained by using the Softmax layer:

[0172] F out = softmax(Linear(F CTA ));

[0173] Here, it can be seen that F fuse in the training directly becomes F CTA , and the preset classification corresponding to the label 1 (i.e., the maximum value of the probability) in F out is determined as the type result of the coronary plaque in the CTA section.

[0174] The method for determining the type of coronary plaque provided in the embodiments of the present application comprises: acquiring a coronary artery image; inputting the coronary artery image into a coronary plaque recognition model to output a type result of the coronary plaque corresponding to the coronary artery image; wherein the coronary plaque recognition model is obtained by training through the following steps: acquiring a coronary artery sample image and a lumen sample image corresponding to the coronary artery sample image from a pre-created sample data set; wherein the coronary artery sample image and the lumen sample image are both unlabeled images; the coronary artery sample image comprises a plurality of blood vessel sections along a blood vessel center line; the lumen sample image comprises a plurality of lumen image sections along a shooting direction; inputting any one of the blood vessel sections in the coronary artery sample image and the plurality of lumen image sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module to obtain a trained image registration training module; obtaining a target lumen image section most matched with each blood vessel section in the coronary artery sample image from the plurality of lumen image sections corresponding to the coronary artery sample image through the trained image registration training module; inputting any one of the blood vessel sections in the coronary artery sample image and the target lumen image section corresponding to the blood vessel section into a plaque recognition training module to train the plaque recognition training module to obtain a trained plaque recognition training module; and determining the trained plaque recognition training module as the coronary plaque recognition model.

[0175] In this way, by using the technical solution provided in the present application, the unlabeled coronary artery sample image (CTA) and the lumen sample image (for example, OCT) are matched, the generalization ability and effect of the model are improved by using large-scale unlabeled data, and the matching result is directly used for plaque recognition in the next stage. In the plaque recognition training stage, the characteristics of the lumen sample image with high resolution (which can identify more and finer types of plaque) are used to guide the model to identify more features in the coronary artery sample image, thereby improving the accuracy of CTA in identifying the type of coronary plaque.

[0176] Based on the same application concept, the embodiments of the present application also provide a device for determining the type of coronary plaque corresponding to the method for determining the type of coronary plaque provided in the above embodiments. Since the principle of solving problems in the device of the embodiments of the present application is similar to that of the method for determining the type of coronary plaque provided in the above embodiments of the present application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described herein.

[0177] Please refer to Figure 5 , Figure 6 , Figure 5 is a structural diagram of a device for determining the type of coronary plaque provided in the embodiments of the present application, Figure 6Figure 2 is a structural diagram of a coronary plaque type determination device provided by an embodiment of the present application. As shown in Figure 2, the determination device 510 comprises: Figure 5

[0178] An acquisition module 511 is configured to acquire a coronary artery image.

[0179] A determination module 512 is configured to input the coronary artery image into a coronary plaque recognition model, and output a type result of a coronary plaque corresponding to the coronary artery image.

[0180] A training module 513 is configured to train the coronary plaque recognition model. The training module 513 comprises an acquisition unit 5131, a first training unit 5132, a matching unit 5133, a second training unit 5134, and a determination unit 5135.

[0181] The acquisition unit 5131 is configured to acquire a coronary artery sample image and an intraluminal sample video corresponding to the coronary artery sample image from a pre-created sample data set. The coronary artery sample image and the intraluminal sample video are both unlabeled images. The coronary artery sample image comprises a plurality of vessel sections along a vessel centerline. The intraluminal sample video comprises a plurality of intraluminal video sections along a shooting direction.

[0182] The first training unit 5132 is configured to input any one of the vessel sections in the coronary artery sample image and the plurality of intraluminal video sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, and obtain a trained image registration training module.

[0183] The matching unit 5133 is configured to obtain, by using the trained image registration training module, a target intraluminal video section that is most matched with each of the vessel sections in the coronary artery sample image from the plurality of intraluminal video sections corresponding to the coronary artery sample image.

[0184] The second training unit 5134 is configured to input any one of the vessel sections in the coronary artery sample image and a target intraluminal video section corresponding to the vessel section into a plaque recognition training module to train the plaque recognition training module, and obtain a trained plaque recognition training module.

[0185] The determination unit 5135 is configured to determine the trained plaque recognition training module as the coronary plaque recognition model.

[0186] Optionally, the first training unit 5132 is specifically configured to:

[0187] ​inputting any one of the vessel sections in the coronary sample images into a first feature extraction layer of the image registration training module to obtain coronary artery features of the vessel section;

[0188] inputting multiple intraluminal image sections in the intraluminal sample images corresponding to the coronary sample images into a second feature extraction layer of the image registration training module to obtain intraluminal image features of each of the intraluminal image sections;

[0189] determining, based on the coronary artery features of the vessel section and the intraluminal image features of each of the intraluminal image sections, a target intraluminal image section that is most matched with the vessel section from the multiple intraluminal image sections;

[0190] determining, based on the target intraluminal image section, a label of each of the intraluminal image sections corresponding to the vessel section;

[0191] obtaining a first loss function based on the label of each of the intraluminal image sections;

[0192] determining whether the first loss function converges;

[0193] if not, updating parameters of the image registration training module, and obtaining a next coronary sample image and intraluminal sample images corresponding to the next coronary sample image to continue training the image registration training module until the first loss function converges;

[0194] if yes, obtaining the trained image registration training module.

[0195] Optionally, the second training unit 5134 is specifically configured to:

[0196] inputting any one of the vessel sections in the coronary sample images and a target intraluminal image section corresponding to the vessel section into a plaque identification training module to extract coronary artery features in the vessel section and intraluminal image features of the target intraluminal image section corresponding to the vessel section;

[0197] obtaining a current influence coefficient, and fusing the coronary artery features in the vessel section and the intraluminal image features of the target intraluminal image section corresponding to the vessel section through the current influence coefficient to obtain fused features;

[0198] after the fused features pass through a full connection layer and normalization processing, obtaining probabilities that a coronary plaque represented by the fused features belongs to each of preset classifications;

[0199] obtaining a second loss function based on the probabilities that the coronary plaque represented by the fused features belongs to each of the preset classifications;

[0200] determining whether the second loss function converges;

[0201] If no, the parameters of the plaque recognition training module and the current influence coefficient are updated, a next coronary artery sample image and a target intraluminal sample image corresponding to the next coronary artery sample image are obtained, and the plaque recognition training module is continuously trained until the second loss function converges.

[0202] If yes, the trained plaque recognition training module is obtained.

[0203] Optionally, the determination module 512 is specifically configured to:

[0204] For each blood vessel section in the coronary artery image, the blood vessel section in the coronary artery image is input into a coronary plaque recognition model for feature extraction, to obtain coronary artery features of the blood vessel section in the coronary artery image;

[0205] After the coronary artery features of the blood vessel section in the coronary artery image are processed through a full connection layer and normalization, probabilities that a coronary plaque represented by the coronary artery features of the blood vessel section in the coronary artery image belongs to each preset classification are obtained.

[0206] In the probabilities, a preset classification corresponding to a probability with the largest value is determined as a type result of the coronary plaque corresponding to the blood vessel section in the coronary artery image.

[0207] Optionally, as shown in Figure 6 The determination apparatus 510 further includes a creation module 514, which is configured to:

[0208] Obtain multiple initial images of a patient obtained through non-invasive coronary artery scanning and an intraluminal sample image of the patient obtained through invasive intraluminal scanning;

[0209] Perform data processing on the multiple initial images of the patient, to obtain a coronary artery sample image of the patient;

[0210] For each patient, the coronary artery sample image of the patient and the intraluminal sample image of the patient are stored as a data pair of the patient;

[0211] After the data pair of each patient is stored, a sample data set is obtained.

[0212] Optionally, when the creation module 514 is used to perform data processing on the multiple initial images of the patient, to obtain the coronary artery sample image of the patient, the creation module 514 is specifically configured to:

[0213] Preprocess the multiple initial images of the patient, to obtain a three-dimensional blood vessel image of the patient;

[0214] segmenting the three-dimensional blood vessel image to obtain a coronary vessel image;

[0215] extracting a blood vessel centerline in the coronary vessel image, and determining a blood vessel cross section perpendicular to the blood vessel centerline along the blood vessel centerline;

[0216] obtaining an image interpolation result of each blood vessel cross section, and splicing the image interpolation result of each blood vessel cross section to obtain a coronary artery sample image of the patient.

[0217] Optionally, the plaque recognition training module comprises a third feature extraction layer and a fourth feature extraction layer; the third feature extraction layer is configured to extract a coronary artery feature in the blood vessel cross section; the fourth feature extraction layer is configured to extract an intraluminal image feature of the target intraluminal image cross section; parameters of the plaque recognition training module comprise parameters of the third feature extraction layer and parameters of the fourth feature extraction layer; and the second training unit 5134 is further configured to:

[0218] use the trained parameters in the first feature extraction layer in the image registration training module as parameters of a network for extracting the coronary artery feature in the blood vessel cross section in the plaque recognition training module, and determine the network with the parameters for extracting the coronary artery feature in the blood vessel cross section as the third feature extraction layer;

[0219] use the trained parameters in the second feature extraction layer in the image registration training module as parameters of a network for extracting the intraluminal image feature of the target intraluminal image cross section in the plaque recognition training module, and determine the network with the parameters for extracting the intraluminal image feature of the target intraluminal image cross section as the fourth feature extraction layer.

[0220] The embodiment of the application provides a kind of determination device of coronary plaque type, the determination device includes: acquisition module, for obtaining coronary artery image;Determination module, for inputing the coronary artery image into coronary plaque identification model, output the type result of coronary plaque corresponding to coronary artery image;Training module, for training coronary plaque identification model;The training module includes acquisition unit, first training unit, matching unit, second training unit and determination unit;The acquisition unit is used to obtain coronary artery sample image and the intraluminal sample image corresponding to the coronary artery sample image from the sample data set created in advance;Wherein, the coronary artery sample image and the intraluminal sample image are all unlabeled images;The coronary artery sample image includes multiple vessel sections along the vessel center line;The intraluminal sample image includes multiple intraluminal image sections along the shooting direction;First training unit, for inputing any one vessel section in the coronary artery sample image and multiple intraluminal image sections corresponding to the coronary artery sample image in the intraluminal sample image into image registration training module to train the image registration training module, obtain trained image registration training module;Matching unit, for obtaining the target intraluminal image section most matched with each vessel section in the coronary artery sample image in multiple intraluminal image sections corresponding to the coronary artery sample image by the trained image registration training module;Second training unit, for inputing any one vessel section in the coronary artery sample image and the target intraluminal image section corresponding to the vessel section into plaque identification training module to train the plaque identification training module, obtain trained plaque identification training module;Determination unit, for determining the trained plaque identification training module as coronary plaque identification model.

[0221] In this way, by using the technical scheme provided by the application, the unlabeled coronary artery sample image (CTA) and the intraluminal sample image (for example, OCT) can be matched, the generalization ability and effect of the model can be improved by using large-scale unlabeled data, and the matching result can be directly used for plaque identification in the next stage. In the plaque identification training stage, the intraluminal sample image is used to identify more and finer types of plaque types, so as to guide the model to identify more features in the coronary artery sample image, and improve the accuracy of CTA in identifying coronary plaque types.

[0222] Please refer to Figure 7 , Figure 7 A structural schematic diagram of an electronic device provided by the embodiment of the application is shown in FIG. 7. Figure 7 As shown in FIG. 7, the electronic device 700 includes a processor 710, a memory 720 and a bus 730.

[0223] The memory 720 stores machine readable instructions executable by the processor 710, when the electronic device 700 is running, the processor 710 and the memory 720 communicate through the bus 730, the machine readable instructions are executed by the processor 710, can execute the steps of the method embodiment as shown in the above Figure 1 and Figure 2 The steps of the determination method of the coronary plaque type in the method embodiment are not repeated here.

[0224] The computer readable storage medium provided in the embodiment of the present application stores a computer program, when the computer program is run by the processor, can execute the steps of the determination method of the coronary plaque type in the method embodiment as shown in the above Figure 1 and Figure 2 The steps of the determination method of the coronary plaque type in the method embodiment are not repeated here.

[0225] The skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0226] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0227] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0228] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0229] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0230] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of determining a type of coronary plaque, characterized by, The determination method comprises: obtaining a coronary artery image; inputting the coronary artery image into a coronary plaque recognition model to output a type result of a coronary plaque corresponding to the coronary artery image; wherein the coronary plaque recognition model is obtained by the following steps: obtaining a coronary artery sample image and a corresponding intraluminal sample image from a pre-created sample data set; wherein the coronary artery sample image and the intraluminal sample image are both unlabeled images; the coronary artery sample image comprises a plurality of blood vessel sections along a blood vessel center line; the intraluminal sample image comprises a plurality of intraluminal image sections along a shooting direction; inputting any one of the blood vessel sections in the coronary artery sample image and a plurality of intraluminal image sections corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, to obtain a trained image registration training module; obtaining a target intraluminal image section most matched with each blood vessel section in the coronary artery sample image from the plurality of intraluminal image sections corresponding to the coronary artery sample image through the trained image registration training module; inputting any one of the blood vessel sections in the coronary artery sample image and the target intraluminal image section corresponding to the blood vessel section into a plaque recognition training module to train the plaque recognition training module, to obtain a trained plaque recognition training module; determining the trained plaque recognition training module as the coronary plaque recognition model; the step of inputting any one of the blood vessel sections in the coronary artery sample image and the target intraluminal image section corresponding to the blood vessel section into the plaque recognition training module to train the plaque recognition training module, to obtain the trained plaque recognition training module, comprises: inputting any one of the blood vessel sections in the coronary artery sample image and the target intraluminal image section corresponding to the blood vessel section into the plaque recognition training module, to extract a coronary artery feature in the blood vessel section and an intraluminal image feature of the target intraluminal image section corresponding to the blood vessel section; obtaining a current influence coefficient, and fusing the coronary artery feature in the blood vessel section and the intraluminal image feature of the target intraluminal image section corresponding to the blood vessel section through the current influence coefficient, to obtain a fusion feature; after the fusion feature is subjected to a full connection layer and normalization processing, obtaining a probability that a coronary plaque represented by the fusion feature belongs to each preset classification; obtaining a second loss function based on the probability that the coronary plaque represented by the fusion feature belongs to each preset classification; determining whether the second loss function converges; if not, updating parameters of the plaque recognition training module and the current influence coefficient, and obtaining a next coronary artery sample image and a target intraluminal sample image corresponding to the next coronary artery sample image to continue training the plaque recognition training module until the second loss function converges; if yes, obtaining the trained plaque recognition training module.

2. The determination method according to claim 1, characterized in that, The step of inputting the blood vessel section in any one of the coronary artery sample images and the plurality of intraluminal image sections in the intraluminal sample image corresponding to the coronary artery sample images into the image registration training module to train the image registration training module to obtain the trained image registration training module comprises: inputting the blood vessel section in any one of the coronary artery sample images into a first feature extraction layer of the image registration training module to obtain the coronary artery feature of the blood vessel section; inputting the plurality of intraluminal image sections in the intraluminal sample image corresponding to the coronary artery sample images into a second feature extraction layer of the image registration training module to obtain the intraluminal image feature of each intraluminal image section; determining the target intraluminal image section most matched with the blood vessel section from the plurality of intraluminal image sections based on the coronary artery feature of the blood vessel section and the intraluminal image feature of each intraluminal image section; determining the label of each intraluminal image section corresponding to the blood vessel section based on the target intraluminal image section; obtaining the first loss function based on the label of each intraluminal image section; determining whether the first loss function converges; if not, updating the parameters of the image registration training module, and obtaining the next coronary artery sample image and the intraluminal sample image corresponding to the next coronary artery sample image to continue training the image registration training module until the first loss function converges; if yes, obtaining the trained image registration training module.

3. The determination method according to claim 1, characterized in that, The step of inputting the coronary artery image into the coronary artery plaque recognition model to output the type result of the coronary artery plaque corresponding to the coronary artery image comprises: for each blood vessel section in the coronary artery image, inputting the blood vessel section in the coronary artery image into the coronary artery plaque recognition model for feature extraction to obtain the coronary artery feature of the blood vessel section in the coronary artery image; after the coronary artery feature of the blood vessel section in the coronary artery image is subjected to a full connection layer and a normalization process, the probability that the coronary artery plaque represented by the coronary artery feature of the blood vessel section in the coronary artery image belongs to each preset classification is obtained; in the probability, the preset classification corresponding to the probability with the maximum value is determined as the type result of the coronary artery plaque corresponding to the blood vessel section in the coronary artery image.

4. The determination method according to claim 1, characterized in that, The sample data set is created by the following steps: obtaining a plurality of initial images of a patient subjected to non-invasive coronary artery scanning and an intraluminal sample image of the patient subjected to invasive intraluminal scanning; performing data processing on the plurality of initial images of the patient to obtain a coronary artery sample image of the patient; for each patient, the coronary artery sample image of the patient and the intraluminal sample image of the patient are stored as a data pair of the patient; after storing the data pair of each patient, a sample data set is obtained.

5. The determination method according to claim 4, characterized in that, The step of performing data processing on the plurality of initial images of the patient to obtain a coronary artery sample image of the patient comprises: performing preprocessing on the plurality of initial images of the patient to obtain a three-dimensional blood vessel image of the patient; performing segmentation processing on the three-dimensional blood vessel image to obtain a coronary artery image; extract a vessel centerline in the coronary vessel image, and determine a vessel cross section perpendicular to the vessel centerline along the vessel centerline; obtain an image interpolation result of each vessel cross section, and splice the image interpolation results of each vessel cross section to obtain a coronary artery sample image of the patient.

6. The determination method of claim 1, wherein, The plaque recognition training module comprises a third feature extraction layer and a fourth feature extraction layer; the third feature extraction layer is configured to extract a coronary artery feature in the vessel cross section; the fourth feature extraction layer is configured to extract an intraluminal image feature of the target intraluminal image section; parameters of the plaque recognition training module comprise parameters of the third feature extraction layer and parameters of the fourth feature extraction layer; The third feature extraction layer and the fourth feature extraction layer are obtained by the following steps: parameters in the first feature extraction layer in the trained image registration training module are taken as parameters of a network for extracting the coronary artery feature in the vessel cross section in the plaque recognition training module, and the network with the parameters for extracting the coronary artery feature in the vessel cross section is determined as the third feature extraction layer; parameters in the second feature extraction layer in the trained image registration training module are taken as parameters of a network for extracting the intraluminal image feature of the target intraluminal image section in the plaque recognition training module, and the network with the parameters for extracting the intraluminal image feature of the target intraluminal image section is determined as the fourth feature extraction layer.

7. An apparatus for determining a type of coronary plaque, characterized by comprising: The determining device comprises: an obtaining module configured to obtain a coronary artery image; a determining module configured to input the coronary artery image into a coronary plaque recognition model, and output a type result of a coronary plaque corresponding to the coronary artery image; a training module configured to train the coronary plaque recognition model; the training module comprises an obtaining unit, a first training unit, a matching unit, a second training unit, and a determining unit; the obtaining unit is configured to obtain a coronary artery sample image and an intraluminal sample image corresponding to the coronary artery sample image from a pre-created sample data set; the coronary artery sample image and the intraluminal sample image are both unlabeled images; the coronary artery sample image comprises a plurality of vessel cross sections along a vessel centerline; the intraluminal sample image comprises a plurality of intraluminal image sections along a shooting direction; the first training unit is configured to input any one of the vessel cross sections in the coronary artery sample image and a plurality of intraluminal image sections in the intraluminal sample image corresponding to the coronary artery sample image into an image registration training module to train the image registration training module, and obtain a trained image registration training module; the matching unit is configured to obtain, by using the trained image registration training module, a target intraluminal image section that is most matched with each of the vessel cross sections in the coronary artery sample image from the plurality of intraluminal image sections corresponding to the coronary artery sample image; the second training unit is configured to input any one of the vessel cross sections in the coronary artery sample image and a target intraluminal image section corresponding to the vessel cross section into a plaque recognition training module to train the plaque recognition training module, and obtain a trained plaque recognition training module. A determination unit is configured to determine the trained plaque recognition training module as a coronary plaque recognition model. The second training unit is specifically configured to: input any one of the vessel sections in the coronary sample images and the target intraluminal image section corresponding to the vessel section into a plaque recognition training module, extract the coronary artery features in the vessel section and the intraluminal image features of the target intraluminal image section corresponding to the vessel section; obtain a current influence coefficient, and fuse the coronary artery features in the vessel section and the intraluminal image features of the target intraluminal image section corresponding to the vessel section through the current influence coefficient to obtain fused features; after the fused features are subjected to a full connection layer and normalization processing, obtain the probability that the coronary plaque represented by the fused features belongs to each preset classification; obtain a second loss function based on the probability that the coronary plaque represented by the fused features belongs to each preset classification; determine whether the second loss function converges; if not, update the parameters of the plaque recognition training module and the current influence coefficient, obtain a next coronary sample image and a target intraluminal sample image corresponding to the next coronary sample image, and continue to train the plaque recognition training module until the second loss function converges; if yes, obtain a trained plaque recognition training module.

8. An electronic device, comprising: The method comprises the following steps: a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the coronary plaque type determination method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the coronary plaque type determination method according to any one of claims 1 to 6.

Images