A blood vessel diameter prediction method and device, an electronic device, and a storage medium

Abstract

The application provides a blood vessel diameter prediction method and device, electronic equipment and storage medium, comprising: acquiring a blood vessel image; for each image block in the blood vessel image, performing feature extraction processing on the image block by a feature extraction module in a diameter prediction model to determine a convolution feature map of the image block; performing down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map by a multi-layer analysis module to determine a first aggregated feature map of the image block; performing lesion prediction on the second aggregated feature map by a lesion correction module to determine a blood vessel lesion probability at a center point of the image block, and correcting the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected correction feature map; and performing diameter prediction processing on the correction feature map by a diameter prediction module to determine a blood vessel diameter at the center point of the image block. In this way, the blood vessel diameter at each position in the blood vessel can be accurately determined by the technical solution of the application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a blood vessel diameter prediction method and device, electronic equipment and storage medium. BACKGROUND

[0002] Coronary angiography is an important method for diagnosing heart disease. In coronary angiography, obtaining the diameter of the blood vessel is an important step, which has the following significance:

[0003] Evaluating the degree of stenosis: By measuring the diameter of the coronary artery, the degree of coronary stenosis can be evaluated. According to the degree of stenosis, the doctor can determine whether to perform interventional treatment, such as stent implantation or coronary artery balloon dilation.

[0004] Planning treatment: The measurement of coronary artery diameter also helps doctors develop treatment plans. For example, in stent implantation surgery, doctors will choose the appropriate stent model and size according to the diameter of the blood vessel.

[0005] Predicting prognosis: The prognosis of coronary artery lesions and the effectiveness of subsequent treatment are also related to the diameter of the blood vessel. Smaller blood vessel diameter is associated with higher risk of coronary heart disease and cardiovascular events.

[0006] Therefore, measuring the diameter of the coronary artery in coronary angiography can provide important information for doctors about the degree of coronary stenosis, treatment planning, and predicting prognosis, so accurate determination of the diameter of the blood vessel in coronary angiography is a technical problem to be solved. SUMMARY

[0007] Therefore, the purpose of the present application is to provide a blood vessel diameter prediction method and device, electronic equipment and storage medium, which can accurately determine the diameter of the blood vessel at each position in the blood vessel, thereby assisting doctors in evaluating the degree of stenosis, planning treatment, and predicting prognosis.

[0008] The present application provides a blood vessel diameter prediction method, which comprises:

[0009] Obtaining a blood vessel image of a target region of a target patient who has undergone image block extraction processing; the blood vessel image includes a plurality of image blocks, and the image blocks are sub-images of a preset size extracted with interval points on a blood vessel center line in the blood vessel image as the center;

[0010] For each image block, input the image block into a pre-trained diameter prediction model, perform feature extraction processing on the image block by a feature extraction module in the diameter prediction model, and determine the convolution feature map of the image block.

[0011] The multi-layer analysis module in the diameter prediction model sequentially performs downsampling, aggregation, weighting, and re-aggregation processing on the convolution feature map to determine a first aggregated feature map of the image block; wherein the first aggregated feature map is a predicted probability of the diameter type determined by a classification submodule in the multi-layer analysis module, a second aggregated feature map obtained by sequentially performing downsampling and aggregation processing on the convolution feature map, a feature map obtained by performing weighting processing and then aggregation processing on the second aggregated feature map;

[0012] The lesion correction module in the diameter prediction model performs lesion prediction on the second aggregated feature map to determine a blood vessel lesion probability at the center point of the image block, and corrects the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected correction feature map.

[0013] The diameter prediction module in the diameter prediction model performs diameter prediction processing on the correction feature map to determine the blood vessel diameter at the center point of the image block.

[0014] Optionally, the image block extraction processing is performed by the following steps:

[0015] Obtaining a blood vessel image of a target region of a target patient; wherein the blood vessel image is a two-dimensional contrast image;

[0016] Performing center line extraction processing on the blood vessel image to determine a blood vessel center line of the blood vessel image formed by a plurality of interval points connected in sequence;

[0017] For each interval point on the center line, performing sub-image extraction processing of a preset size centered on the interval point to obtain a plurality of image blocks.

[0018] Optionally, the diameter prediction model is constructed by the following steps:

[0019] Obtaining a plurality of training blood vessel images that have undergone image block extraction processing; wherein each training image block in each training blood vessel image is added with a true label;

[0020] Sequentially inputting each training image block of each training blood vessel image into an initial diameter prediction model to determine a prediction result of the blood vessel at the center of the training image block;

[0021] Comparing the prediction result with the true label to determine a target loss function value;

[0022] Based on the target loss function value, iteratively training the initial diameter prediction model and updating the model parameters of the initial diameter prediction model until the target loss function value converges, stopping training, and obtaining the diameter prediction model.

[0023] Optionally, the real labels comprise a diameter type label, a lesion binary classification label and a diameter numerical value label, and the comparing the prediction results with the real labels to determine a target loss function value comprises:

[0024] comparing a diameter type prediction result in the prediction results with the diameter type label to determine a first loss function value;

[0025] comparing a lesion prediction result in the prediction results with the lesion binary classification label to determine a second loss function value;

[0026] comparing a diameter numerical value prediction result in the prediction results with the diameter numerical value label to determine a third loss function value;

[0027] weighting and summing the first loss function value, the second loss function value and the third loss function value to determine the target loss function value.

[0028] Optionally, the sequentially performing, by the multi-layer analysis module in the diameter prediction model, down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map to determine a first aggregated feature map of the image block comprises:

[0029] performing, by a down-sampling sub-module in the multi-layer analysis module, multiple times of down-sampling processing on the convolution feature map to determine multiple down-sampled feature maps of the image block;

[0030] performing, by an aggregation sub-module in the multi-layer analysis module, aggregation processing on the multiple down-sampled feature maps to determine a second aggregated feature map of the image block;

[0031] performing, by a classification sub-module in the multi-layer analysis module, diameter type prediction on the second aggregated feature map to determine a prediction probability of each diameter type, and performing weighting processing on a down-sampled feature map corresponding to each diameter type using the determined prediction probability of each diameter type to obtain multiple weighted feature images;

[0032] performing, by the aggregation sub-module, aggregation processing on the multiple weighted feature images to determine the first aggregated feature map of the image block.

[0033] Optionally, before performing the weighting processing on the down-sampled feature map corresponding to each diameter type using the determined prediction probability of each diameter type, the prediction method further comprises:

[0034] performing up-sampling processing on each down-sampled feature map to make the size of each processed down-sampled feature map the same as the size of the image block.

[0035] Optionally, the blood vessel lesion probability is used to correct the first aggregated feature map to obtain a corrected correction feature map, including:

[0036] The blood vessel lesion probability is multiplied with the first aggregated feature map to obtain a third aggregated feature map;

[0037] The third aggregated feature map is added to the first aggregated feature map, and a feature map obtained after the addition is determined as the correction feature map.

[0038] Embodiments of the present application also provide a blood vessel diameter prediction device, which comprises:

[0039] An acquisition unit is configured to acquire a blood vessel image of a target region of a target patient which has undergone image block extraction processing; the blood vessel image comprises a plurality of image blocks, and each image block is a preset size of a sub-image extracted with an interval point on a blood vessel center line in the blood vessel image as a center;

[0040] A feature extraction unit is configured to input each image block into a pre-trained diameter prediction model, perform feature extraction processing on the image block by a feature extraction module in the diameter prediction model, and determine a convolution feature map of the image block;

[0041] A multi-layer analysis unit is configured to sequentially perform downsampling, aggregation, weighting and re-aggregation processing on the convolution feature map by a multi-layer analysis module in the diameter prediction model, and determine a first aggregated feature map of the image block; wherein the first aggregated feature map is a feature map obtained by performing weighting processing on a second aggregated feature map obtained by sequentially performing downsampling and aggregation processing on the convolution feature map, and then performing re-aggregation processing; the second aggregated feature map is a predicted probability of a diameter type determined by a classification module in the multi-layer analysis module;

[0042] A correction unit is configured to perform lesion prediction on the second aggregated feature map by a lesion correction module in the diameter prediction model, determine a blood vessel lesion probability at a center point of the image block, and use the determined blood vessel lesion probability to correct the first aggregated feature map to obtain a corrected correction feature map;

[0043] A prediction unit is configured to perform diameter prediction on the correction feature map by a diameter prediction module in the diameter prediction model, and determine a blood vessel diameter at the center point of the image block.

[0044] Optionally, the prediction device further comprises an image block extraction unit, which is configured to:

[0045] Acquire a blood vessel image at a target region of a target patient; wherein the blood vessel image is a two-dimensional contrast image;

[0046] performing center line extraction processing on the blood vessel image to determine a blood vessel center line of the blood vessel image formed by a plurality of interval points being sequentially connected;

[0047] For each interval point on the center line, performing sub-image extraction processing of a preset size centered on the interval point to obtain a plurality of image blocks.

[0048] Optionally, the prediction device further comprises a model construction unit, which is configured to:

[0049] obtain a plurality of training blood vessel images on which image block extraction processing has been performed; wherein each training image block in each training blood vessel image is added with a true label;

[0050] sequentially input each training image block of each training blood vessel image into the initial diameter prediction model to determine a prediction result of the blood vessel at the center of the training image block;

[0051] compare the prediction result with the true label to determine a target loss function value;

[0052] based on the target loss function value, iteratively train the initial diameter prediction model and update the model parameters of the initial diameter prediction model until the target loss function value converges, stop training, and obtain the diameter prediction model.

[0053] Optionally, the true label comprises a diameter type label, a lesion binary classification label, and a diameter numerical value label, and when the model construction unit is configured to compare the prediction result with the true label to determine a target loss function value, the model construction unit is configured to:

[0054] compare the diameter type prediction result in the prediction result with the diameter type label to determine a first loss function value;

[0055] compare the lesion prediction result in the prediction result with the lesion binary classification label to determine a second loss function value;

[0056] compare the diameter numerical value prediction result in the prediction result with the diameter numerical value label to determine a third loss function value;

[0057] weight and sum the first loss function value, the second loss function value, and the third loss function value to determine the target loss function value.

[0058] Optionally, when the multi-layer analysis unit is configured to determine the first aggregated feature map of the image block by sequentially performing down-sampling, aggregation, weighting and re-aggregation on the convolution feature map by the multi-layer analysis module in the diameter prediction model, the multi-layer analysis unit is configured to:

[0059] performing multiple times of down-sampling on the convolution feature map by a down-sampling sub-module in the multi-layer analysis module to determine multiple down-sampled feature maps of the image block;

[0060] performing aggregation on the multiple down-sampled feature maps by an aggregation sub-module in the multi-layer analysis module to determine a second aggregated feature map of the image block;

[0061] performing diameter type prediction on the second aggregated feature map by a classification sub-module in the multi-layer analysis module to determine a prediction probability of each diameter type, and performing weighting on the down-sampled feature map corresponding to each diameter type using the determined prediction probability of each diameter type to obtain multiple weighted feature maps;

[0062] performing aggregation on the multiple weighted feature maps by the aggregation sub-module to determine the first aggregated feature map of the image block.

[0063] Optionally, before performing weighting on the down-sampled feature map corresponding to each diameter type using the determined prediction probability of each diameter type, the multi-layer analysis unit is further configured to:

[0064] performing up-sampling on each down-sampled feature map to make the size of each processed down-sampled feature map the same as the size of the image block.

[0065] Optionally, when the correction unit is configured to correct the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected feature map, the correction unit is configured to:

[0066] multiply the blood vessel lesion probability with the first aggregated feature map to obtain a third aggregated feature map;

[0067] add the third aggregated feature map and the first aggregated feature map to determine the feature map obtained after addition as the corrected feature map.

[0068] Embodiments of the present application also provide an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the prediction method as described above.

[0069] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is run by a processor to perform the steps of the prediction method.

[0070] The embodiment of the present application provides a blood vessel diameter prediction method and device, an electronic device and a storage medium. The method comprises the following steps: obtaining a blood vessel image of a target region of a target patient which has undergone image block extraction processing; the blood vessel image comprises a plurality of image blocks, the image blocks are sub-images of a preset size extracted with interval points on a blood vessel center line in the blood vessel image as centers; for each image block, the image block is input into a pre-trained diameter prediction model, a feature extraction module in the diameter prediction model performs feature extraction processing on the image block, and a convolution feature map of the image block is determined; a multi-layer analysis module in the diameter prediction model sequentially performs down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map, and a first aggregated feature map of the image block is determined; the first aggregated feature map is a prediction probability of a diameter type determined by a classification submodule in the multi-layer analysis module, a second aggregated feature map obtained by sequentially performing down-sampling and aggregation processing on the convolution feature map, and a feature map obtained by performing weighting processing and then aggregation processing on the second aggregated feature map; a lesion correction module in the diameter prediction model performs lesion prediction on the second aggregated feature map, determines a blood vessel lesion probability at a center point of the image block, and corrects the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected correction feature map; and a diameter prediction module in the diameter prediction model performs diameter prediction processing on the correction feature map, and determines a blood vessel diameter at the center point of the image block.

[0071] In this way, the technical solution provided by the present application introduces a multi-layer analysis module in the diameter prediction model according to the difference in blood vessel diameter distribution, introduces a lesion correction module in the diameter prediction model according to the difference between healthy blood vessels and lesions, thereby improving the accuracy of the blood vessel diameter prediction result, and further assisting doctors in stenosis degree evaluation, treatment plan planning and prognosis prediction.

[0072] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0074] Figure 1 Flow chart of a blood vessel diameter prediction method provided by the embodiments of the present application;

[0075] Figure 2 Flow chart of a diameter prediction model training process provided by the present application;

[0076] Figure 3 Structure diagram of a diameter prediction model provided by the present application;

[0077] Figure 4 Structure diagram of a blood vessel diameter prediction device provided by the embodiments of the present application;

[0078] Figure 5 Structure diagram of a blood vessel diameter prediction device provided by the embodiments of the present application;

[0079] Figure 6 Structure diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by those skilled in the art without creative labor belongs to the scope of the present application.

[0081] Coronary angiography is an important method for diagnosing heart disease, and in coronary angiography, obtaining blood vessel diameter is an important step, which has the following significance:

[0082] Assessing the degree of stenosis: By measuring the diameter of the coronary artery, the degree of coronary stenosis can be assessed. According to the degree of stenosis, the doctor can determine whether to perform interventional treatment, such as stent implantation or coronary balloon dilatation.

[0083] Planning treatment: Measurement of coronary artery diameter also helps doctors develop treatment plans. For example, in stent implantation surgery, doctors will choose the appropriate stent model and size according to the diameter of the blood vessel.

[0084] Predicting prognosis: The prognosis of coronary artery lesions and the effectiveness of subsequent treatment are also related to the diameter of the blood vessel. Smaller blood vessel diameter is associated with a higher risk of coronary heart disease and cardiovascular events.

[0085] Therefore, measuring the diameter of the coronary artery in coronary angiography can provide doctors with important information about the degree of coronary stenosis, treatment planning, and predicting prognosis, so accurate determination of the diameter of the blood vessel in coronary angiography is a technical problem to be solved.

[0086] Based on this, the embodiments of the present application provide a blood vessel diameter prediction method, device, electronic equipment and storage medium, which can accurately determine the diameter of the blood vessel at each position in the blood vessel, thereby assisting doctors in stenosis assessment, treatment planning and predicting prognosis.

[0087] Please refer to Figure 1 , Figure 1 The flowchart of a blood vessel diameter prediction method provided by the embodiments of the present application. As shown in Figure 1 The prediction method provided by the embodiments of the present application comprises:

[0088] S101, acquiring a blood vessel image of a target region of a target patient which has undergone image block extraction processing.

[0089] Here, the blood vessel image includes a plurality of image blocks, and the image blocks are sub-images of a preset size extracted with interval points on a blood vessel center line in the blood vessel image as the center. Wherein, the size of each image block is the same, and the number of image blocks is the same as the number of interval points on the center line.

[0090] Wherein, the target region is not limited to the heart, brain and abdomen in the human body. For example, when the target region is the heart region, the blood vessel image is a coronary blood vessel image.

[0091] In an embodiment provided by the present application, the image block extraction processing is performed by the following steps:

[0092] Step 11, acquiring a blood vessel image of a target region of a target patient.

[0093] Here, the blood vessel image is a two-dimensional angiogram.

[0094] Step 12, performing center line extraction processing on the blood vessel image to determine the blood vessel center line of the blood vessel image formed by a plurality of interval points connected in turn.

[0095] In this step, the following center line extraction methods are included but not limited to:

[0096] Method 1: Threshold-based binarization method: convert the two-dimensional image into a binary image, then use morphological processing, edge detection and other algorithms to obtain the blood vessel contour, and finally use the skeleton extraction algorithm to extract the blood vessel center line.

[0097] Method 2: Image segmentation-based method: use region-based image segmentation algorithm to separate blood vessels and background in the image, then use morphological processing, edge detection and other algorithms to extract the blood vessel contour, and finally use the skeleton extraction algorithm to extract the blood vessel center line.

[0098] Method 3: Machine learning-based method: use machine learning algorithm, train a large number of blood vessel images to get the model of blood vessels, and then use the model to segment and extract the center line of the new blood vessel image.

[0099] Method 4: Filter-based method: use different filtering algorithms, such as Hessian matrix-based filtering, Frangi filtering, etc., to get the response value of the blood vessel, and then use the blood vessel response value to extract the blood vessel contour and center line.

[0100] Step 13, for each interval point on the center line, a sub-image extraction process of a preset size is performed with the interval point as the center, and a plurality of image blocks are obtained.

[0101] Here, the size of the image block is predetermined.

[0102] For example, when an 80x80 image block needs to be extracted, it can be extracted according to the following formula.

[0103]

[0104] Where (x, y) represents the coordinates of the current interval point on the center line in the image coordinate system, and the colon: represents the starting point from the value on the left of the colon, and the value is taken one by one until the value on the right of the colon as the end.

[0105] S102, for each image block, input the image block into the pre-trained diameter prediction model, and use the feature extraction module in the diameter prediction model to perform feature extraction processing on the image block to determine the convolution feature map of the image block.

[0106] Here, the diameter prediction model is a model pre-trained for predicting blood vessel diameter, and the diameter prediction model mainly includes a feature extraction module, a multi-layer analysis module, a lesion correction module and a diameter prediction module.

[0107] The feature extraction module includes a plurality of first convolution sub-modules, each first convolution sub-module being composed of a convolution layer, a BN layer and a ReLU layer.

[0108]

[0109] The input feature map (image block) is represented by, and the ReLU layer can be represented by:

[0110]

[0111] The BN layer can be represented by:

[0112]

[0113] The input feature map (image block) is represented by, and the ReLU layer can be represented by: and are learnable scaling and shifting parameters for rescaling and shifting the normalized result back into the network. and are the mean and variance of the features corresponding to all samples in a batch, is a very small constant to prevent the variance from being 0. For example, the convolution kernel (Conv311) used in the convolution layer in this module has a size of 3x3, a stride of 1, and a padding of 1, so the size of the feature map after this module is the same as that of the input image block. It should be noted that the size of the convolution and, the stride and the padding can also be selected as other values, as long as the size of the feature map is the same as that of the input image block.

[0114] In an embodiment provided in the present application, the diameter prediction model is constructed by the following steps:

[0115] Step 21, obtaining a plurality of training blood vessel images which have been subjected to image block extraction processing.

[0116] Here, a plurality of training blood vessel images are obtained, each training blood vessel image being an image which has been subjected to image block extraction processing, each training blood vessel image including a plurality of training image blocks, and each training image block being added with a real label.

[0117] The real label includes a diameter type label, a lesion binary classification label and a diameter numerical label.

[0118] ​​​The diameter type is classified according to the numerical value of the diameter. For example, a diameter greater than 2 mm is determined as a type, a diameter greater than 1 mm but less than 2 mm is determined as a type, and a diameter less than or equal to 1 mm is determined as another type. The selection of the diameter threshold is not limited to 1 mm and 2 mm, and can be selected as needed, which is not limited herein. In this way, when adding the diameter numerical value label to the to-be-trained image block, the diameter type label can be automatically generated.

[0119] It should be noted that the diameter of the coronary artery varies from person to person, and the specific numerical value also varies with the position and branches. Generally speaking, the diameter of the left coronary artery (LAD) is about 2-4 mm, the diameter of the left circumflex branch (LCX) is about 1.5-2.5 mm, and the diameter of the right coronary artery (RCA) is about 2-3.5 mm. However, these numerical values are only an average value, and there are great differences between individuals. This does not include some branch vessels. Therefore, the design of the diameter prediction model in the present scheme introduces a classification module in the multi-layer analysis module, which more fully considers the above complex situations.

[0120] Step 22, sequentially input each to-be-trained image block of each to-be-trained blood vessel image into the initial diameter prediction model to determine the prediction result of the blood vessel at the center of the to-be-trained image block.

[0121] Here, the prediction result includes a diameter type prediction result, a lesion prediction result, and a diameter numerical value prediction result.

[0122] Step 23, comparing the prediction result with the true label to determine a target loss function value.

[0123] Here, each to-be-trained image block determines a target loss function value.

[0124] In one embodiment, the comparison of the prediction result with the true label to determine the target loss function value includes: comparing the diameter type prediction result in the prediction result with the diameter type label to determine a first loss function value; comparing the lesion prediction result in the prediction result with the lesion binary classification label to determine a second loss function value; comparing the diameter numerical value prediction result in the prediction result with the diameter numerical value label to determine a third loss function value; and weighting and summing the first loss function value, the second loss function value, and the third loss function value to determine the target loss function value.

[0125] Here, the diameter type prediction result is determined by the multi-layer analysis module in the initial diameter prediction model, the lesion prediction result is determined by the lesion correction module in the initial diameter prediction model, and the diameter numerical value prediction result is determined by the diameter prediction model in the initial diameter prediction model.

[0126] The weight coefficients of the first loss function value, the second loss function value and the third loss function value can be set to 1 or other coefficients, which are not limited herein.

[0127] Step 24, based on the target loss function value, iteratively training the initial diameter prediction model and updating the model parameters of the initial diameter prediction model until the target loss function value converges, stopping training, and obtaining the diameter prediction model.

[0128] Here, the model parameters of the initial diameter prediction model are updated once after each target loss function value is determined. The model parameters of the initial diameter prediction model can be updated by a back propagation algorithm.

[0129] For example, please refer to Figure 2 , Figure 2 A flowchart of a diameter prediction model training process provided by the present application is shown in FIG. 1. As shown in FIG. 1, the above modules are trained in a joint manner, and the output of the model has three parts, which are diameter type prediction results, lesion prediction results and diameter value prediction results. For example, the loss of the diameter type prediction results and the diameter type label adopts a multi-class CrossEntropy loss function, the loss of the lesion prediction results and the lesion binary classification label adopts a binary CrossEntropy loss function, and the loss of the diameter value prediction results and the diameter value label adopts an L1 loss function. Figure 2

[0130]

[0131] wherein , , respectively represent the true labels of classification, lesion and diameter, respectively represent the prediction results of classification, lesion and diameter. Different batches of training data are input into the network model, and then the loss function value Loss is calculated by comparing the prediction results with the true labels. In the training process, the Loss value is continuously reduced until it no longer decreases, and then the training is completed to obtain the diameter prediction model.

[0132] S103, sequentially performing down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map by a multi-layer analysis module in the diameter prediction model to determine a first aggregated feature map of the image block.

[0133] ​Here, the first aggregated feature map is a feature map obtained by performing weighting processing on a second aggregated feature map obtained by performing sequential downsampling and aggregation processing on a convolutional feature map, and performing aggregation processing again.

[0134] The multi-layer analysis module includes a downsampling submodule, an aggregation submodule, and a classification submodule.

[0135] In an embodiment, the first aggregated feature map of the image block is determined by sequentially performing downsampling, aggregation, weighting, and aggregation again on the convolutional feature map by the multi-layer analysis module in the diameter prediction model, including:

[0136] S1031, the convolutional feature map is processed by the downsampling submodule in the multi-layer analysis module multiple times, and a plurality of downsampled feature maps of the image block are determined.

[0137] Here, the downsampling submodule includes a plurality of second convolutional submodules, each of which can output a downsampled feature map, and the input of each second convolutional submodule is the output of the previous module. Each second convolutional submodule performs one downsampling processing on the convolutional feature map. For example, the input of the second second convolutional submodule is the downsampled feature map of the output of the first second convolutional submodule.

[0138] The downsampling submodule and the feature extraction module are similar and are both composed of a plurality of convolutional submodules, but the parameters of the convolutional layers are different. For example, the downsampling submodule includes three second convolutional submodules, each of which is composed of a convolutional layer, a BN layer, and a ReLU layer. The layer parameters of the convolutional layer are Conv 321 , which means that the convolution kernel size is 3x3, the step is 2, and the padding is 1. Therefore, after passing through each second convolutional submodule, the feature map size is reduced by half (assuming the original size is 80x80, and the size after passing through three second convolutional submodules is 80x80—>40x40—>20x20). The three levels of convolutional modules correspond to:

[0139]

[0140] wherein, is the diameter of the blood vessel, represents a feature map with a feature size of 80x80, and Similarly, here, it should be noted that The reason is that the small size blood vessels are more superficial, and the receptive field is smaller and the collected information is more local, while the other two are more deep and the collected information is more extensive.

[0141] S1032、by the aggregation sub-module in the multi-layer analysis module, the plurality of down-sampling feature maps are aggregated to determine the second aggregated feature map of the image block.

[0142] Here, before the plurality of down-sampling feature maps are aggregated by the aggregation sub-module in the multi-layer analysis module, the plurality of down-sampling feature maps need to be up-sampled to make the size of each down-sampling image after processing the same as the size of the convolution feature map.

[0143] Continue the example in S1031, by the aggregation sub-module, { , , } are aggregated together, here the up-sampling method is used to up-sample to the same size as :

[0144]

[0145] Wherein represents concatenating features along their channel direction, such as concat({F 128×80×80 , F 64×80×80}) = F (128+64)×80×80 , and Up(F, N) represents N times up-sampling of the feature F, which is performed by linear interpolation, such as Up(F 40 , 2) = F 80 .

[0146] S1033, by the classification sub-module in the multi-layer analysis module, the second aggregated feature map is used to predict the diameter type, determine the prediction probability of each diameter type, and use the determined prediction probability of each diameter type to respectively weight the down-sampling feature map corresponding to the diameter type, to obtain a plurality of weighted feature images.

[0147] For example, assume that the diameter type is divided into the following three categories: the diameter greater than 2mm is determined as a class, the diameter greater than 1mm but less than 2mm is determined as a class, and the diameter less than or equal to 1mm is determined as another class. The prediction of the diameter type is:

[0148]

[0149] Wherein, represents the predicted classification probability value Pc∈R 1×3For example, [0.1, 0.7, 0.2], the sum of the vector is 1, which represents the probability value of each channel, for example, 0.1 represents the probability of output diameter d>2mm is 0.1, 0.7 represents the probability of output diameter greater than 1mm but less than 2mm is 0.7, and 0.2 represents the probability of output diameter less than or equal to 1mm is 0.2.

[0150] In an implementation, before the prediction method uses the determined prediction probability of each diameter type to respectively weight the down-sampling feature map corresponding to the diameter type, the prediction method further comprises: respectively up-sampling each down-sampling feature map, so that the size of each processed down-sampling feature map is the same as the size of the image block.

[0151] In this way, the prediction method uses the determined prediction probability of each diameter type to respectively weight the down-sampling feature map corresponding to the diameter type, and obtains a plurality of weighted feature images, which can be understood by the following formula:

[0152]

[0153] wherein, represents the probability value of the first channel, that is, 0.1 in the above example, represents the probability value of the second channel, that is, 0.7 in the above example, represents the probability value of the third channel, that is, 0.2 in the above example.

[0154] S1034, the aggregation submodule aggregates the plurality of weighted feature images to determine a first aggregated feature map of the image block.

[0155] For example, the aggregation method corresponding to the first aggregated feature map is illustrated by the following formula:

[0156]

[0157] wherein, represents the first aggregated feature map output after aggregation.

[0158] S104, the lesion correction module in the diameter prediction model performs lesion prediction on the second aggregated feature map to determine the probability of vascular lesion at the center point of the image block, and corrects the first aggregated feature map using the determined probability of vascular lesion to obtain a corrected feature map.

[0159] Here, the lesion correction module can be composed of a plurality of Linear layers, for example, the lesion correction module in the present scheme can be composed of two Linear layers.

[0160] In an embodiment, the first aggregated feature map is corrected using the determined blood vessel lesion probability to obtain a corrected feature map, including: multiplying the blood vessel lesion probability and the first aggregated feature map to obtain a third aggregated feature map; adding the third aggregated feature map and the first aggregated feature map to determine the feature map obtained after addition as the corrected feature map.

[0161] For example, the first aggregated feature map is corrected using the determined blood vessel lesion probability to obtain a corrected feature map, and the corresponding formula expression is as follows:

[0162]

[0163] Wherein, is the corrected feature map, is the first aggregated feature map, is the blood vessel lesion probability, .

[0164] S105, the diameter prediction module in the diameter prediction model performs diameter prediction processing on the corrected feature map to determine the blood vessel diameter at the center point of the image block.

[0165] Here, the corrected feature map obtained after correction by the lesion correction module is input into the diameter prediction module in the diameter prediction model, and the diameter prediction module performs diameter prediction processing on the corrected feature map to determine the blood vessel diameter value at the center point of the image block.

[0166] In this way, after the diameter prediction model determines the blood vessel diameter value at the center point of each image block, the diameters of the blood vessels at different positions in the blood vessel image can be determined according to the order of the image blocks.

[0167] For example, please refer to Figure 3 , Figure 3 The structure diagram of the diameter prediction model provided in the present application is shown in Figure 3 The number of the first convolutional sub-module and the second convolutional sub-module in the present scheme is selected as 3, and other numbers can also be selected, which can be selected according to actual conditions. It should be noted that Figure 3 The two aggregation sub-modules shown are the same, and Figure 3 The two aggregation sub-modules are shown in order to better show the data flow process of the data processing process, so the multi-layer analysis module only includes one aggregation sub-module.

[0168] Thus, the technical scheme provided by the present application introduces a multi-layer analysis module in the diameter prediction model according to the difference in the diameter distribution of the blood vessels, and introduces a lesion correction module in the diameter prediction model according to the difference between the healthy blood vessels and the diseased blood vessels, thereby improving the accuracy of the blood vessel diameter prediction result, and further assisting the doctor in the stenosis degree evaluation, the treatment plan planning and the prognosis prediction.

[0169] Please refer to Figure 4 、 Figure 5 , Figure 4 FIG. 1 is a structural schematic diagram of a blood vessel diameter prediction device provided by an embodiment of the present application, Figure 5 FIG. 2 is another structural schematic diagram of a blood vessel diameter prediction device provided by an embodiment of the present application. As shown in FIG. 2, the prediction device 400 comprises: Figure 4

[0170] An acquisition unit 410 is configured to acquire a blood vessel image of a target region of a target patient which has undergone image block extraction processing; the blood vessel image comprises a plurality of image blocks, and each image block is a sub-image of a preset size extracted with an interval point on a blood vessel center line in the blood vessel image as a center;

[0171] A feature extraction unit 420 is configured to input each image block into a pre-trained diameter prediction model, perform feature extraction processing on the image block by a feature extraction module in the diameter prediction model, and determine a convolution feature map of the image block;

[0172] A multi-layer analysis unit 430 is configured to sequentially perform down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map by a multi-layer analysis module in the diameter prediction model, and determine a first aggregated feature map of the image block; wherein the first aggregated feature map is a feature map obtained by performing weighting processing on a second aggregated feature map obtained by sequentially performing down-sampling and aggregation processing on the convolution feature map, and then performing aggregation processing again; and the prediction probability of the diameter type of the first aggregated feature map is determined by a classification module in the multi-layer analysis module;

[0173] A correction unit 440 is configured to perform lesion prediction on the second aggregated feature map by a lesion correction module in the diameter prediction model, determine a blood vessel lesion probability at a center point of the image block, and correct the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected correction feature map;

[0174] A prediction unit 450 is configured to perform diameter prediction on the correction feature map by a diameter prediction module in the diameter prediction model, and determine a blood vessel diameter at the center point of the image block.

[0175] Optionally, as shown in FIG. 3, Figure 5 ​As shown, the prediction device 400 further comprises an image block extraction unit 460, which is configured to:

[0176] obtain a blood vessel image at a target region of a target patient; wherein the blood vessel image is a two-dimensional contrast image;

[0177] perform a centerline extraction process on the blood vessel image to determine a blood vessel centerline of the blood vessel image formed by a plurality of interval points connected in sequence;

[0178] for each interval point on the centerline, perform a sub-image extraction process of a preset size centered on the interval point to obtain a plurality of image blocks.

[0179] Optionally, the prediction device 400 further comprises a model construction unit 470, which is configured to:

[0180] obtain a plurality of training blood vessel images that have undergone image block extraction processing; wherein each training image block in each training blood vessel image is added with a true label;

[0181] input each training image block of each training blood vessel image into an initial diameter prediction model in sequence to determine a prediction result of a blood vessel at the center of the training image block;

[0182] compare the prediction result with the true label to determine a target loss function value;

[0183] based on the target loss function value, iteratively train the initial diameter prediction model and update model parameters of the initial diameter prediction model until the target loss function value converges, stop training, and obtain the diameter prediction model.

[0184] Optionally, the true label comprises a diameter type label, a lesion binary classification label, and a diameter numerical value label, and when the model construction unit 470 is configured to compare the prediction result with the true label to determine a target loss function value, the model construction unit 470 is configured to:

[0185] compare a diameter type prediction result in the prediction result with the diameter type label to determine a first loss function value;

[0186] compare a lesion prediction result in the prediction result with the lesion binary classification label to determine a second loss function value;

[0187] compare a diameter numerical value prediction result in the prediction result with the diameter numerical value label to determine a third loss function value;

[0188] The target loss function value is determined by weighted sum of the first loss function value, the second loss function value and the third loss function value.

[0189] Optionally, when the first aggregated feature map of the image block is determined by sequentially performing down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map by the multi-layer analysis module in the diameter prediction model, the multi-layer analysis unit 430 is configured to:

[0190] performing multiple times of down-sampling processing on the convolution feature map by a down-sampling sub-module in the multi-layer analysis module to determine multiple down-sampled feature maps of the image block;

[0191] performing aggregation processing on the multiple down-sampled feature maps by an aggregation sub-module in the multi-layer analysis module to determine a second aggregated feature map of the image block;

[0192] performing diameter type prediction on the second aggregated feature map by a classification sub-module in the multi-layer analysis module to determine a prediction probability of each diameter type, and performing weighting processing on the down-sampled feature map corresponding to the diameter type by using the determined prediction probability of each diameter type respectively to obtain multiple weighted feature images;

[0193] performing aggregation processing on the multiple weighted feature images by the aggregation sub-module to determine the first aggregated feature map of the image block.

[0194] Optionally, before performing weighting processing on the down-sampled feature map corresponding to the diameter type by using the determined prediction probability of each diameter type respectively, the multi-layer analysis unit 430 is further configured to:

[0195] performing up-sampling processing on each down-sampled feature map respectively so that the size of each processed down-sampled feature map is the same as the size of the image block.

[0196] Optionally, when the first aggregated feature map is modified to obtain a modified feature map by using the determined blood vessel lesion probability, the modification unit 440 is configured to:

[0197] multiplying the blood vessel lesion probability and the first aggregated feature map to obtain a third aggregated feature map;

[0198] adding the third aggregated feature map and the first aggregated feature map to determine the feature map obtained after addition as the modified feature map.

[0199] Please refer to Figure 6 , Figure 6 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the electronic device includes a processor 100, a memory 200, a communication interface 300 and a bus 400. Figure 6As shown in FIG. 6, the electronic device 600 includes a processor 610, a memory 620 and a bus 630.

[0200] The memory 620 stores machine readable instructions executable by the processor 610, and when the electronic device 600 is running, the processor 610 communicates with the memory 620 through the bus 630. When the machine readable instructions are executed by the processor 610, the above-mentioned Figure 1 and Figure 2 The steps in the method embodiment shown above can be executed by the processor 610, and the specific implementation manner can refer to the method embodiment, which will not be described here.

[0201] The application embodiment further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps in the method embodiment shown above can be executed, and the specific implementation manner can refer to the method embodiment, which will not be described here. Figure 1 and Figure 2 The steps in the method embodiment shown above can be executed by the processor 610, and the specific implementation manner can refer to the method embodiment, which will not be described here.

[0202] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0203] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, and for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0204] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0205] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0206] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0207] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of predicting a blood vessel diameter, characterized by, The prediction method comprises: obtaining a blood vessel image of a target region of a target patient which has undergone image block extraction processing; the blood vessel image comprises a plurality of image blocks, and each image block is a preset size sub-image extracted with an interval point on a blood vessel center line in the blood vessel image as a center; for each image block, inputting the image block into a pre-trained diameter prediction model, performing feature extraction processing on the image block by a feature extraction module in the diameter prediction model, and determining a convolution feature map of the image block; performing, by a multi-layer analysis module in the diameter prediction model, sequentially downsampling, aggregation, weighting and re-aggregation processing on the convolution feature map, and determining a first aggregated feature map of the image block; wherein the first aggregated feature map is a prediction probability of a diameter type determined by a classification submodule in the multi-layer analysis module, a second aggregated feature map obtained by sequentially performing downsampling and aggregation processing on the convolution feature map, a feature map obtained by performing weighting processing on the second aggregated feature map and then performing aggregation processing; performing lesion prediction on the second aggregated feature map by a lesion correction module in the diameter prediction model, determining a blood vessel lesion probability at a center point of the image block, and correcting the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected correction feature map; performing diameter prediction processing on the correction feature map by a diameter prediction module in the diameter prediction model, and determining a blood vessel diameter at the center point of the image block; The diameter prediction model is constructed by the following steps: obtaining a plurality of training blood vessel images which have undergone image block extraction processing; wherein each training image block in each training blood vessel image is added with a true label; sequentially inputting each training image block of each training blood vessel image into an initial diameter prediction model to determine a prediction result of a blood vessel at the center of the training image block; comparing the prediction result with the true label to determine a target loss function value; based on the target loss function value, iteratively training the initial diameter prediction model and updating model parameters of the initial diameter prediction model until the target loss function value converges, stopping training, and obtaining the diameter prediction model.

2. The prediction method of claim 1, wherein, The image block extraction processing is performed by the following steps: obtaining a blood vessel image at a target region of a target patient; wherein the blood vessel image is a two-dimensional contrast image; performing center line extraction processing on the blood vessel image to determine a blood vessel center line of the blood vessel image formed by a plurality of interval points connected in sequence; for each interval point on the center line, performing preset size sub-image extraction processing with the interval point as a center to obtain a plurality of image blocks.

3. The prediction method of claim 1, wherein, The true label comprises a diameter type label, a lesion binary classification label and a diameter numerical value label, and the comparison of the prediction result with the true label to determine a target loss function value comprises: comparing a diameter type prediction result in the prediction result with the diameter type label to determine a first loss function value; comparing a lesion prediction result in the prediction result with the lesion binary classification label to determine a second loss function value; The diameter numerical value prediction result in the prediction result is compared with the diameter numerical value label to determine a third loss function value; The first loss function value, the second loss function value and the third loss function value are weighted and summed to determine the target loss function value.

4. The prediction method of claim 1, wherein, The first aggregated feature map of the image block is determined by sequentially performing down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map by the multi-layer analysis module in the diameter prediction model, comprising: The plurality of down-sampled feature maps of the image block are determined by performing multiple down-sampling processing on the convolution feature map by the down-sampling submodule in the multi-layer analysis module; The second aggregated feature map of the image block is determined by performing aggregation processing on the plurality of down-sampled feature maps by the aggregation submodule in the multi-layer analysis module; The prediction probability of each diameter type is determined by performing diameter type prediction on the second aggregated feature map by the classification submodule in the multi-layer analysis module, and the down-sampled feature map corresponding to the diameter type is weighted by using the determined prediction probability of each diameter type, to obtain a plurality of weighted feature images; The first aggregated feature map of the image block is determined by performing aggregation processing on the plurality of weighted feature images by the aggregation submodule.

5. The prediction method of claim 4, wherein, Before the down-sampled feature map corresponding to the diameter type is weighted by using the determined prediction probability of each diameter type, the prediction method further comprises: Each down-sampled feature map is up-sampled to make the size of each processed down-sampled feature map the same as the size of the image block.

6. The prediction method of claim 1, wherein, The first aggregated feature map is modified by using the determined blood vessel lesion probability to obtain a modified feature map, comprising: The third aggregated feature map is obtained by multiplying the blood vessel lesion probability and the first aggregated feature map; The modified feature map is determined by adding the third aggregated feature map and the first aggregated feature map.

7. A blood vessel diameter prediction device characterized by comprising: The prediction device comprises: An acquisition unit is configured to acquire a blood vessel image of a target region of a target patient that has undergone image block extraction processing; the blood vessel image includes a plurality of image blocks, and each image block is a sub-image of a preset size extracted with an interval point on a blood vessel center line in the blood vessel image as a center; A feature extraction unit is configured to input each image block into a pre-trained diameter prediction model, and perform feature extraction processing on the image block by a feature extraction module in the diameter prediction model to determine a convolution feature map of the image block; A multi-layer analysis unit is configured to sequentially perform down-sampling, aggregation, weighting and re-aggregation processing on the convolution feature map by a multi-layer analysis module in the diameter prediction model to determine a first aggregated feature map of the image block; wherein the first aggregated feature map is obtained by weighting a second aggregated feature map obtained by sequentially performing down-sampling and aggregation processing on the convolution feature map, and then performing aggregation processing on the second aggregated feature map by using a prediction probability of a diameter type determined by a classification module in the multi-layer analysis module. The correction unit is configured to perform lesion prediction on the second aggregated feature map by a lesion correction module in the diameter prediction model, determine a blood vessel lesion probability at a center point of the image block, and correct the first aggregated feature map using the determined blood vessel lesion probability to obtain a corrected correction feature map; The prediction unit is configured to perform diameter prediction on the corrected feature map by a diameter prediction module in the diameter prediction model, and determine a blood vessel diameter at the center point of the image block. The prediction device further comprises a model construction unit, which is configured to: obtain a plurality of to-be-trained blood vessel images that have undergone image block extraction processing; wherein each to-be-trained image block in each to-be-trained blood vessel image is added with a real label; input each to-be-trained image block of each to-be-trained blood vessel image into an initial diameter prediction model in sequence to determine a prediction result of a blood vessel at the center of the to-be-trained image block; compare the prediction result with the real label to determine a target loss function value; based on the target loss function value, iteratively train the initial diameter prediction model and update model parameters of the initial diameter prediction model until the target loss function value converges, stop training, and obtain the diameter prediction model.

8. An electronic device, comprising: comprise: a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to perform the steps of the prediction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the prediction method according to any one of claims 1 to 6.

Images