Vascular calcification degree determination method and device, computer equipment and storage medium

Abstract

The invention relates to a vascular calcification degree determination method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an image frame sequence containing a plurality of image frames; an image segmentation model is adopted to segment the plurality of image frames, image segmentation results corresponding to the plurality of image frames are obtained, and each image segmentation result comprises an intima segmentation area and a calcification segmentation area; the image segmentation model is obtained through training based on sample coding images corresponding to a plurality of sample image frames; each sample coding image is obtained by coding the current sample image frame according to a plurality of previous sample image frames and a plurality of later sample image frames of the corresponding current sample image frame; and determining the vascular calcification degree according to the intimal segmentation areas and the calcification segmentation areas corresponding to the plurality of image frames. By adopting the method, the accuracy of the determined vascular calcification degree can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly to a method, device, computer device, storage medium, and computer program product for determining the degree of vascular calcification. Background Art

[0002] Traditional determination of the degree of vascular calcification often uses image segmentation algorithms or coronary angiography methods. Among them, traditional image segmentation algorithms segment calcified plaques in intravascular ultrasound images and are easily affected by image quality. For example, the image is not clear, or there is a lot of noise, resulting in an unsatisfactory image segmentation result. In addition, coronary angiography has a low diagnostic sensitivity for intravascular calcified plaques and cannot provide information about calcified plaques. Therefore, traditional methods for determining the degree of vascular calcification have the problem of low accuracy in determining the degree of vascular calcification. Summary of the Invention

[0003] Based on this, it is necessary to provide a method, device, computer device, computer-readable storage medium, and computer program product for determining the degree of vascular calcification, which can improve the accuracy of the determined degree of vascular calcification.

[0004] In a first aspect, the present application provides a method for determining the degree of vascular calcification, including:

[0005] Obtain an image frame sequence including a plurality of image frames;

[0006] Use an image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the respective image frames, where each image segmentation result includes an intimal segmentation region and a calcified segmentation region; the image segmentation model is trained based on sample encoded images corresponding to the respective sample image frames; each sample encoded image is obtained by encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames corresponding to the current sample image frame;

[0007] Determine the degree of vascular calcification according to the intimal segmentation regions and calcified segmentation regions corresponding to the respective image frames.

[0008] In one of the embodiments, using the image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the respective image frames includes:

[0009] For each image frame, perform sharpening processing on the plurality of previous image frames, the plurality of subsequent image frames, and the current image frame respectively, and sum the pixel values of the pixel points at the same position in the obtained plurality of sharpened processing results to obtain a first encoded image corresponding to the current image frame;

[0010] Use the current image frame as the second encoded image corresponding to the current image frame;

[0011] Average the pixel values of the pixels at the same positions in multiple previous image frames, multiple subsequent image frames, and the current image frame to obtain a first intermediate image; Subtract the pixel values of the pixels at the same positions in the first intermediate image and the current image frame to obtain a second intermediate image; Normalize the second intermediate image to obtain the third encoded image corresponding to the current image frame;

[0012] Use the first encoded image, the second encoded image, and the third encoded image corresponding to the current image frame as the encoded image corresponding to the current image frame;

[0013] Use an image segmentation model to separately segment the encoded images corresponding to multiple image frames to obtain the image segmentation results corresponding to multiple image frames.

[0014] In one embodiment, determining the degree of vascular calcification based on the intima segmentation regions and calcification segmentation regions corresponding to multiple image frames includes:

[0015] Reconstruct a three-dimensional model of the target blood vessel based on the intima segmentation regions and calcification segmentation regions corresponding to multiple image frames;

[0016] Determine the calcification region in the three-dimensional model and determine the shape parameters of the calcification region;

[0017] Determine the degree of vascular calcification based on the shape parameters and a preset evaluation strategy.

[0018] In one embodiment, reconstructing a three-dimensional model of the target blood vessel based on the intima segmentation regions and calcification segmentation regions corresponding to multiple image frames includes:

[0019] For each image frame, use an adventitia segmentation model to segment the current image frame to obtain the adventitia segmentation result corresponding to the current image frame, and the adventitia segmentation result includes the adventitia segmentation region;

[0020] According to the adventitia segmentation region corresponding to the current image frame, respectively correct the intima segmentation region and the calcification segmentation region corresponding to the current image frame to obtain the intima correction region and the calcification correction region corresponding to the current image frame;

[0021] Determine the scanning order of multiple image frames, and based on the scanning order, perform three-dimensional reconstruction on the adventitia segmentation regions, intima correction regions, and calcification correction regions corresponding to multiple image frames to obtain a three-dimensional model of the target blood vessel.

[0022] In one embodiment, according to the outer membrane segmentation region corresponding to the current image frame, the inner membrane segmentation region and the calcification segmentation region corresponding to the current image frame are respectively corrected to obtain the inner membrane correction region and the calcification correction region corresponding to the current image frame, including:

[0023] Taking the part of the inner membrane segmentation region corresponding to the current image frame that is within the outer membrane segmentation region corresponding to the current image frame as the inner membrane correction region corresponding to the current image frame;

[0024] Taking the part of the calcification segmentation region corresponding to the current image frame that is within the outer membrane segmentation region corresponding to the current image frame as the calcification correction region corresponding to the current image frame.

[0025] In one embodiment, the shape parameters include the calcification region angle and the calcification region length; based on the shape parameters and a preset evaluation strategy, determining the degree of vascular calcification includes:

[0026] Determining a calcification angle score according to the calcification region angle and a preset angle;

[0027] Determining a calcification length score according to the calcification region length and a preset length;

[0028] Performing a weighted sum of the calcification angle score and the calcification length score to obtain a summation result;

[0029] Querying a preset mapping relationship to determine the degree of vascular calcification corresponding to the summation result.

[0030] In one embodiment, the training steps of the image segmentation model include:

[0031] Obtaining a plurality of sample image frames obtained by scanning a sample blood vessel and a plurality of labeled segmentation images respectively corresponding to the plurality of sample image frames, each labeled segmentation image including an inner membrane labeled region and a calcification labeled region;

[0032] For each sample image frame, encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames of the current sample image frame to obtain a sample encoded image corresponding to the current sample image frame;

[0033] Inputting the sample encoded image corresponding to the current sample image frame into an initial segmentation model to obtain a predicted segmentation image corresponding to the current sample image frame, the predicted segmentation image including an inner membrane prediction region and a calcification prediction region;

[0034] Inputting the predicted segmentation image corresponding to the current sample image frame into an initial discriminant model to obtain a first probability of whether the predicted segmentation image is a generated image, and inputting the labeled segmentation image corresponding to the current sample image frame into the initial discriminant model to obtain a second probability of whether the labeled segmentation image is a generated image,

[0035] Based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to multiple sample image frames, a model loss is calculated.

[0036] According to the model loss, the model parameters of the initial segmentation model and the initial discrimination model are updated until the model loss meets a preset stopping condition, and an image segmentation model is obtained.

[0037] In one embodiment, calculating the model loss based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to multiple sample image frames includes:

[0038] For each sample image frame, according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame, the segmentation model loss corresponding to the current sample image frame is determined.

[0039] According to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame, the first discrimination loss corresponding to the current sample image frame is determined. According to the first probability corresponding to the current sample image frame and the second probability corresponding to the current sample image frame, the second discrimination loss corresponding to the current sample image frame is determined. According to the first discrimination loss corresponding to the current sample image frame and the second discrimination loss corresponding to the current sample image frame, the discrimination model loss corresponding to the current sample image frame is determined.

[0040] According to the segmentation model losses and discrimination model losses corresponding to multiple sample image frames, the model loss is calculated.

[0041] In a second aspect, the present application further provides a device for determining the degree of vascular calcification, including:

[0042] An acquisition module, configured to acquire an image frame sequence including multiple image frames;

[0043] A segmentation module, configured to separately segment multiple image frames by using an image segmentation model to obtain image segmentation results corresponding to multiple image frames, and each image segmentation result includes an intima segmentation region and a calcification segmentation region; the image segmentation model is trained based on sample encoded images corresponding to multiple sample image frames; each sample encoded image is obtained by encoding the current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame;

[0044] A determination module, configured to determine the degree of vascular calcification according to the intima segmentation regions and calcification segmentation regions corresponding to multiple image frames.

[0045] In a third aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Obtain an image frame sequence including a plurality of image frames;

[0047] Use an image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the respective plurality of image frames. Each image segmentation result includes an intima segmentation region and a calcification segmentation region. The image segmentation model is trained based on sample encoded images corresponding to the respective plurality of sample image frames. Each sample encoded image is obtained by encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames corresponding to the current sample image frame. Determine the degree of vascular calcification according to the intima segmentation regions and calcification segmentation regions corresponding to the respective plurality of image frames.

[0048] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0049] Obtain an image frame sequence including a plurality of image frames;

[0050] Use an image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the respective plurality of image frames. Each image segmentation result includes an intima segmentation region and a calcification segmentation region. The image segmentation model is trained based on sample encoded images corresponding to the respective plurality of sample image frames. Each sample encoded image is obtained by encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames corresponding to the current sample image frame.

[0051] Determine the degree of vascular calcification according to the intima segmentation regions and calcification segmentation regions corresponding to the respective plurality of image frames.

[0052] In a fifth aspect, the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0053] Obtain an image frame sequence including a plurality of image frames;

[0054] Use an image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the respective plurality of image frames. Each image segmentation result includes an intima segmentation region and a calcification segmentation region. The image segmentation model is trained based on sample encoded images corresponding to the respective plurality of sample image frames. Each sample encoded image is obtained by encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames corresponding to the current sample image frame.

[0055] Determine the degree of vascular calcification based on the intimal segmentation region and the calcification segmentation region corresponding to each of multiple image frames.

[0056] The above method, device, computer device, storage medium, and computer program product for determining the degree of vascular calcification obtain an image frame sequence including multiple image frames, and use an image segmentation model to separately segment the multiple image frames to obtain image segmentation results corresponding to each of the multiple image frames. Each image segmentation result includes an intimal segmentation region and a calcification segmentation region. The image segmentation model is trained based on the sample encoded images corresponding to each of the multiple sample image frames. Each sample encoded image is obtained by encoding the current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame. Since the textures in different regions of blood vessels have certain differences, encoding based on multiple previous and subsequent sample image frames, the sample encoded images incorporate the temporal features of multiple sample image frames, which is beneficial to improving the contrast between the intima and calcified plaques in blood vessels. Training the image segmentation model based on the sample encoded images corresponding to each of the multiple sample image frames and using the image segmentation model to separately segment the multiple image frames is beneficial to improving the accuracy of the image segmentation results. Determining the degree of vascular calcification based on the intimal segmentation region and the calcification segmentation region in the image segmentation results is beneficial to improving the accuracy of the determined degree of vascular calcification. Description of the Drawings

[0057] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0058] Figure 1 It is an application environment diagram of the method for determining the degree of vascular calcification in an embodiment;

[0059] Figure 2 It is a flowchart of the method for determining the degree of vascular calcification in an embodiment;

[0060] Figure 3 It is a schematic diagram of the method for three-dimensional reconstruction in an embodiment;

[0061] Figure 4 It is a schematic diagram of the calcified region in the three-dimensional model in an embodiment;

[0062] Figure 5 It is a schematic diagram of the image frame of the target blood vessel in an embodiment;

[0063] Figure 6Schematic diagram of the training process of an image segmentation model in an embodiment;

[0064] Figure 7 Schematic flowchart of a method for determining the degree of vascular calcification in another embodiment;

[0065] Figure 8 Schematic diagram of the model structures of an image segmentation model and an adventitia segmentation model in an embodiment;

[0066] Figure 9 Block diagram of the structure of a device for determining the degree of vascular calcification in an embodiment;

[0067] Figure 10 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0068] In order to make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] The method for determining the degree of vascular calcification provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 where the terminal 102 communicates with the server 104 through a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed in the cloud or other network servers. The terminal 102 obtains a plurality of image frame sequences including image frames obtained by scanning a target blood vessel; for each image frame, encodes the current image frame according to a plurality of previous image frames and a plurality of subsequent image frames of the current image frame to obtain an encoded image corresponding to the current image frame; uses an image segmentation model to segment the encoded image corresponding to the current image frame to obtain an image segmentation result corresponding to the current image frame, and each image segmentation result includes an intima segmentation region and a calcification segmentation region; determines the degree of vascular calcification according to the intima segmentation regions and the calcification segmentation regions respectively corresponding to the plurality of image frames. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0070] In an exemplary embodiment, as shown in Figure 2 a method for determining the degree of vascular calcification is provided, and this method is applied to Figure 1Taking the terminal 102 in [as an example for illustration, the following steps 202 to 206 are included. Among them:

[0071] Step 202, obtain an image frame sequence including multiple image frames.

[0072] Among them, the image frames can be obtained by scanning a target blood vessel. The target blood vessel can be a blood vessel containing calcified plaques. In some embodiments, the device for scanning the target blood vessel can be an ultrasound scanning device. For example, an IVUS (Intravenous Ultrasound) device. The image frame sequence is obtained by converting the ultrasound signals collected by the IVUS device within one retraction cycle. Since during the process of scanning the target blood vessel, the IVUS device has different reception times for the ultrasound signals returned from different positions in the target blood vessel, therefore, the multiple image frames in the converted image frame sequence have a chronological order.

[0073] Step 204, use an image segmentation model to segment each of the multiple image frames respectively to obtain an image segmentation result corresponding to each of the multiple image frames. Each image segmentation result includes an intima segmentation region and a calcification segmentation region; the image segmentation model is trained based on the sample encoded images corresponding to multiple sample image frames; each sample encoded image is obtained by encoding the current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame.

[0074] Among them, the image segmentation model is a trained deep learning model for image segmentation. The sample image frames are obtained by scanning a sample blood vessel. The terminal obtains a sample image frame sequence including multiple sample image frames obtained by scanning the sample blood vessel.

[0075] The multiple previous sample image frames are any multiple image frames in the sample image frame sequence that are before the current sample image frame. For example, they can be two adjacent image frames before the current sample image frame. The multiple subsequent sample image frames are any multiple image frames in the sample image frame sequence that are after the current sample image frame. For example, they can be two adjacent image frames after the current sample image frame. The number of the multiple previous sample image frames and the number of the multiple subsequent sample image frames can be the same or different.

[0076] Encoding the current sample image frame refers to the process of encoding and combining the current sample image frame, multiple previous sample image frames, and multiple subsequent sample image frames. Since there is a chronological order among the multiple sample image frames, the sample encoded image obtained thus fuses the image features and chronological features corresponding to the current sample image frame, multiple previous sample image frames, and multiple subsequent sample image frames respectively.

[0077] The terminal uses the sample encoded images corresponding to multiple sample image frames as training samples to train a machine learning model, obtaining an image segmentation model. Since the sample blood vessels include calcified plaques and vascular intima, the sample image frames obtained by scanning the sample blood vessels may include the vascular intima and calcified plaques. Since the vascular intima and calcified plaques have different texture features, the sample encoded images fuse the image features and temporal features of multiple front and back sample image frames. The image segmentation model trained based on the sample encoded images can identify the intima region and calcified region in the image frame, generating an image segmentation result corresponding to the image frame. The intima segmentation region in the image segmentation result is used to indicate the vascular intima of the target blood vessel, and the calcified segmentation region in the image segmentation result is used to indicate the calcified plaques of the target blood vessel.

[0078] Step 206: Determine the degree of blood vessel calcification according to the intima segmentation regions and calcified segmentation regions corresponding to multiple image frames.

[0079] Among them, the degree of blood vessel calcification is used to characterize the coverage degree of calcified plaques in the blood vessel. The larger the area occupied by the calcified plaques in the target blood vessel, the higher the corresponding degree of blood vessel calcification.

[0080] Integrating the intima segmentation regions and calcified segmentation regions corresponding to multiple image frames to determine the degree of blood vessel calcification is beneficial to improving the accuracy of the determined degree of blood vessel calcification.

[0081] In the above method for determining the degree of blood vessel calcification, by obtaining an image frame sequence of multiple image frames, using the image segmentation model to separately segment multiple image frames, obtaining the image segmentation results corresponding to multiple image frames, each image segmentation result including an intima segmentation region and a calcified segmentation region, the image segmentation model is trained based on the sample encoded images corresponding to multiple sample image frames, and each sample encoded image is obtained by encoding the current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame. Since the textures of different regions in the blood vessel have certain differences, encoding based on multiple front and back sample image frames, the sample encoded images fuse the temporal features of multiple sample image frames, which is beneficial to improving the contrast between the intima and calcified plaques in the blood vessel. Training the image segmentation model based on the sample encoded images corresponding to multiple sample image frames and using the image segmentation model to separately segment multiple image frames is beneficial to improving the accuracy of the image segmentation results. Determining the degree of blood vessel calcification based on the intima segmentation region and calcified segmentation region in the image segmentation result is beneficial to improving the accuracy of the determined degree of blood vessel calcification.

[0082] In an exemplary embodiment, an image segmentation model is used to segment multiple image frames respectively, obtaining image segmentation results corresponding to each of the multiple image frames, including: for each image frame, performing sharpening processing on multiple previous image frames, multiple subsequent image frames, and the current image frame respectively, summing the pixel values of pixel points at the same position in the multiple obtained sharpening processing results, obtaining a first encoded image corresponding to the current image frame; taking the current image frame as the second encoded image corresponding to the current image frame; averaging the pixel values of pixel points at the same position in the multiple previous image frames, multiple subsequent image frames, and the current image frame, obtaining a first intermediate image; taking the difference between the pixel values of pixel points at the same position in the first intermediate image and the current image frame, obtaining a second intermediate image; performing normalization processing on the second intermediate image, obtaining a third encoded image corresponding to the current image frame; taking the first encoded image, second encoded image, and third encoded image corresponding to the current image frame as the encoded image corresponding to the current image frame; using the image segmentation model to segment the encoded images corresponding to each of the multiple image frames respectively, obtaining image segmentation results corresponding to each of the multiple image frames.

[0083] Among them, the sharpening processing refers to compensating the contour of the image frame, enhancing the edges and the parts with gray level jumps of the image frame, making the image frame clearer. For example, the Laplacian sharpening processing method can be used for the sharpening processing. The multiple sharpening processing results include multiple sharpened previous image frames, multiple sharpened subsequent image frames, and the sharpened current image frame. Since the sizes of the multiple image frames are the same, the sizes of the corresponding multiple sharpening processing results are the same. For the multiple sharpening processing results, the pixel values of pixel points at the same position in the terminal are summed to obtain the first encoded image corresponding to the current image frame.

[0084] In some embodiments, the sharpening processing is represented by S(I), and the current image frame is represented by I n I n-1 I n-2 respectively represent the previous image frames of the current image frame, and I n+1 I n+2 respectively represent the previous image frames and subsequent image frames of the current image frame, then the first encoded image can be represented by the following formula:

[0085] Since the texture of blood is in a scattered and continuously disordered change, after sharpening the multiple previous and subsequent image frames, it is beneficial to enhance the texture features of blood in the image frame, which is beneficial to capturing the blood vessel edges in the image frame and beneficial to identifying the intima of blood vessels.

[0086] The second encoded image of the current image frame is the current image frame itself. In some embodiments, the second encoded image is represented by and can be expressed as:

[0087] The first intermediate image is the pixel average value of the pixel points at the same position in multiple front and rear image frames.

[0088] The second intermediate image is the pixel difference between the first intermediate image and the pixel points at the same position in the current image frame.

[0089] Normalization refers to converting the pixel value of each pixel point in the second intermediate image into a gray value between 0 and 255. In some embodiments, the normalization method may be: for the pixel value of each pixel point in the second intermediate image, subtracting the current pixel value from the minimum pixel value in the second intermediate image to obtain a difference value, dividing the obtained difference value by the maximum pixel value in the second intermediate image to obtain a quotient value, and multiplying the obtained quotient value by 255. The obtained result is used as the current normalized pixel value corresponding to the current pixel value. From the normalized pixel values corresponding to each pixel point in the second intermediate image, the third encoded image of the current image frame is determined. The third encoded image of the current image frame is used to characterize the degree of difference between the current image frame and the pixel average value of multiple front and rear image frames.

[0090] In some embodiments, Norm [0,255] represents normalization to a gray value between 0 and 255, then the third encoded image can be represented by the following formula:

[0091]

[0092] In some embodiments, the encoded image corresponding to the current image frame is a three-channel image, and each channel is the first encoded image, the second encoded image, and the third encoded image respectively.

[0093] In traditional image segmentation methods, the image segmentation model directly segments the image frame to obtain the image segmentation result. In the embodiments of the present application, it is proposed to use the image segmentation model to separately segment the encoded images corresponding to multiple image frames to obtain the image segmentation results corresponding to multiple image frames respectively. For the image frames scanned by an intravascular ultrasound scanning device, the texture of blood vessels is relatively static, and the texture of blood in blood vessels mostly shows scattered dots and continuously changes disorderly. Through the texture change differences in different regions of multiple consecutive image frames, the blood vessel edges can be captured. At the same time, the calcified plaques in the target blood vessels are usually small and difficult to be recognized. Through the guidance of multiple consecutive image frames, the calcified plaque regions can be determined. Therefore, segmenting based on the encoded image is beneficial to improving the accuracy of the image segmentation result.

[0094] In this embodiment, multiple methods are used to encode each image frame, which is beneficial to enhancing the texture features of blood and calcified plaques in the image frame, thereby facilitating the capture of the blood vessel edge in the image frame, the identification of the vascular intima, and the identification of calcified plaque features.

[0095] In an exemplary embodiment, the degree of vascular calcification is determined according to the intima segmentation region and the calcification segmentation region corresponding to each of the multiple image frames, including: reconstructing a three-dimensional model of the target blood vessel according to the intima segmentation region and the calcification segmentation region corresponding to each of the multiple image frames; determining the calcified region in the three-dimensional model and determining the shape parameters of the calcified region; and determining the degree of vascular calcification based on the shape parameters and a preset evaluation strategy.

[0096] Among them, the intima segmentation region and the calcification segmentation region corresponding to each of the multiple image frames, the three-dimensional model of the target blood vessel obtained through three-dimensional reconstruction, and the three-dimensional model includes an intima region and a calcification region.

[0097] The calcified region is the region in the three-dimensional model corresponding to the calcification segmentation region, and the terminal determines the shape parameters of the calcified region. The shape parameters may include parameters such as length, angle, and area. The shape parameters of the calcified region are used to characterize the shape and size of the calcified plaque in the target blood vessel, and the degree of vascular calcification is used to characterize the coverage degree of the calcified plaque in the blood vessel. The preset evaluation strategy is used to characterize the mapping relationship between the shape parameters and the degree of vascular calcification. By looking up the preset evaluation strategy, the degree of vascular calcification corresponding to the shape parameters can be determined.

[0098] In this embodiment, the intima segmentation region and the calcification segmentation region obtained by segmenting the image frame through the image segmentation model can reconstruct a relatively accurate three-dimensional model of the target blood vessel, and then, according to the shape parameters of the calcified region in the three-dimensional model, the degree of vascular calcification is evaluated and determined, which is beneficial to obtaining a relatively accurate degree of vascular calcification.

[0099] In an exemplary embodiment, reconstructing a three-dimensional model of the target blood vessel according to the intima segmentation region and the calcification segmentation region corresponding to each of the multiple image frames includes: for each image frame, using an adventitia segmentation model to segment the current image frame to obtain the adventitia segmentation result corresponding to the current image frame, and the adventitia segmentation result includes an adventitia segmentation region; correcting the intima segmentation region and the calcification segmentation region corresponding to the current image frame respectively according to the adventitia segmentation region corresponding to the current image frame to obtain the intima correction region and the calcification correction region corresponding to the current image frame; determining the scanning order of the multiple image frames, and based on the scanning order, performing three-dimensional reconstruction on the adventitia segmentation regions, intima correction regions, and calcification correction regions corresponding to each of the multiple image frames to obtain a three-dimensional model of the target blood vessel.

[0100] Among them, the adventitia segmentation model is a trained deep learning model. In some embodiments, the adventitia segmentation model may have the same or different model structures as the image segmentation model, and the adventitia segmentation model may use the same or different model training methods as the image segmentation model.

[0101] Since the texture of the vascular adventitia is relatively static, the adventitia part in each image frame can be segmented separately. In some embodiments, the adventitia segmentation model can be directly used to segment each image frame to obtain the adventitia segmentation result corresponding to each image frame. In other embodiments, each image frame can also be optimized, and the adventitia segmentation model is used to segment each optimized image frame to obtain the adventitia segmentation result corresponding to each image frame. The optimization methods include gamma correction and non-uniform filtering, and the optimization is used to suppress image noise and retain image edges.

[0102] In some embodiments, three-dimensional reconstruction can be performed according to the adventitia segmentation regions, intima segmentation regions, and calcification segmentation regions corresponding to multiple image frames to obtain a three-dimensional model of the target blood vessel.

[0103] In other embodiments, the adventitia segmentation regions can also be used to correct the corresponding intima segmentation regions and calcification segmentation regions respectively, so as to perform three-dimensional reconstruction based on the adventitia segmentation regions, intima correction regions, and calcification correction regions corresponding to multiple image frames to obtain a three-dimensional model of the target blood vessel.

[0104] The scanning order refers to the order of receiving ultrasonic signals after performing ultrasonic scanning on the target blood vessel, and the receiving order of the ultrasonic signals is the same as the order of generating the corresponding image frames. Aligning and stitching multiple image frames in the scanning order is beneficial to obtaining a more accurate three-dimensional model of the target blood vessel.

[0105] Such as Figure 3 shown is a schematic diagram of the three-dimensional reconstruction method in an embodiment. For each image frame, the terminal obtains the adventitia region edge of the adventitia segmentation region, the intima region edge of the intima correction region, and the calcification region edge of the calcification correction region in the current image frame, and reconstructs a three-dimensional model of the target blood vessel from the adventitia region edges, intima region edges, and calcification region edges corresponding to multiple image frames.

[0106] In this embodiment, the adventitia segmentation region is segmented by the adventitia segmentation model, and the adventitia segmentation region is used to correct the intima segmentation region and the calcification segmentation region, which is beneficial to improving the accuracy of the segmentation result. Further, three-dimensional reconstruction based on the adventitia segmentation region, intima correction region, and calcification correction region is beneficial to improving the accuracy of the three-dimensional model of the target blood vessel.

[0107] In an exemplary embodiment, according to the adventitia segmentation region corresponding to the current image frame, the intima segmentation region and the calcification segmentation region corresponding to the current image frame are respectively corrected to obtain the intima correction region and the calcification correction region corresponding to the current image frame, including: taking the part of the intima segmentation region corresponding to the current image frame that is within the adventitia segmentation region corresponding to the current image frame as the intima correction region corresponding to the current image frame; taking the part of the calcification segmentation region corresponding to the current image frame that is within the adventitia segmentation region corresponding to the current image frame as the calcification correction region corresponding to the current image frame.

[0108] In this embodiment, since calcified plaques in blood vessels often lie between the intima and the adventitia of the blood vessel, therefore, the adventitia segmentation region can be used to correct the intima segmentation region and the calcification segmentation region, so as to remove the parts of the intima segmentation region and the calcification segmentation region that exceed the adventitia segmentation region, and only take the intersection region as the intima correction region and the calcification correction region, which is beneficial to improving the accuracy of the reconstructed three-dimensional model.

[0109] In an exemplary embodiment, the shape parameters include the calcification region angle and the calcification region length; based on the shape parameters and a preset evaluation strategy, determining the degree of blood vessel calcification includes: determining a calcification angle score according to the calcification region angle and a preset angle; determining a calcification length score according to the calcification region length and a preset length; performing a weighted sum of the calcification angle score and the calcification length score to obtain a summation result; querying a preset mapping relationship to determine the degree of blood vessel calcification corresponding to the summation result.

[0110] Among them, as Figure 4 shown is a schematic diagram of the calcification region in the three-dimensional model in the embodiment. Among them, the calcification region angle is the maximum angle from the center of the blood vessel to the calcification region in the three-dimensional model of the target blood vessel. The calcification region length is the maximum length of the calcification region in the three-dimensional model of the target blood vessel.

[0111] In some embodiments, when the calcification region angle is greater than the preset angle, determining the calcification angle score as a first score; when the calcification region angle is less than or equal to the preset angle, determining the calcification angle score as a second score; when the calcification region length is greater than the preset length, determining the calcification length score as a third score; when the calcification region length is less than or equal to the preset length, determining the calcification length score as a fourth score.

[0112] In some embodiments, the preset evaluation strategy table stores the mapping relationship between the shape parameters of the calcification region and the score. As shown in Table 1 is the preset evaluation strategy table.

[0113] Table 1 Preset Evaluation Strategy Table

[0114]

[0115] The obtained calcification angle score and calcification length score are weighted and summed, and the summation result is the blood vessel calcification degree score.

[0116] The preset mapping relationship stores the mapping relationship between the blood vessel calcification degree score and the blood vessel calcification degree. Query the preset mapping relationship to obtain the blood vessel calcification degree corresponding to the blood vessel calcification degree score.

[0117] In some embodiments, the blood vessel calcification degree includes severe, general, normal, etc. In the case where the blood vessel calcification degree is severe, more aggressive lesion treatment measures are required. In the case where the blood vessel calcification degree is general, experts need to further determine the lesion.

[0118] In this embodiment, through the calcification angle score corresponding to the angle of the calcification region and the calcification length score corresponding to the length of the calcification region, the two scores are weighted and summed. The preset mapping relationship stores the mapping relationship between the blood vessel calcification degree score and the blood vessel calcification degree. According to the summation result, the corresponding blood vessel calcification degree can be directly determined, improving the diagnosis efficiency of calcified plaques, facilitating the reduction of the time required for diagnosis, and reducing the burden on doctors.

[0119] In an exemplary embodiment, the training steps of the image segmentation model include: obtaining a plurality of sample image frames obtained by scanning a sample blood vessel and the corresponding labeled segmentation images for each of the plurality of sample image frames, each labeled segmentation image including an intima labeled region and a calcification labeled region; for each sample image frame, encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames of the current sample image frame to obtain a sample encoded image corresponding to the current sample image frame; inputting the sample encoded image corresponding to the current sample image frame into an initial segmentation model to obtain a predicted segmentation image corresponding to the current sample image frame, the predicted segmentation image including an intima prediction region and a calcification prediction region; inputting the predicted segmentation image corresponding to the current sample image frame into an initial discrimination model to obtain a first probability of whether the predicted segmentation image is a generated image, inputting the labeled segmentation image corresponding to the current sample image frame into the initial discrimination model to obtain a second probability of whether the labeled segmentation image is a generated image, and calculating a model loss based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to the plurality of sample image frames; updating the model parameters of the initial segmentation model and the initial discrimination model according to the model loss until the model loss meets a preset stop condition to obtain the image segmentation model.

[0120] Among them, multiple sample image frames are obtained by scanning a sample blood vessel, and the sample blood vessel may include calcified plaques. The intima and calcified plaques of the sample image frames are labeled. The pixel values of the pixel points corresponding to the labeled intima are set to the first pixel value, the pixel values of the pixel points corresponding to the labeled calcified plaques are set to the second pixel value, and the pixel values of the pixel points that are neither the intima nor the calcified plaques are set to the third pixel value, so as to obtain the labeled segmentation image corresponding to the sample image frame. The pixel values of the pixel points in the intima labeling area and the calcified labeling area are different.

[0121] Since multiple sample image frames are obtained by scanning a sample blood vessel, each sample image frame has a scanning sequence. According to the scanning sequence, multiple previous sample image frames of the current sample image frame and multiple subsequent sample image frames of the current sample image frame are determined among the multiple sample image frames.

[0122] Using the encoding method as in step 204, the current sample image frame is encoded according to the multiple previous sample image frames and multiple subsequent sample image frames of the current sample image frame, so as to obtain the sample encoded image corresponding to the current sample image frame.

[0123] The initial segmentation model is a deep learning model for classification. For example, it can be a U-Net (U-shaped convolutional neural network) structure or a SwinUNETR (Unet-like structure) structure. Among them, the SwinUNETR structure is a segmentation network that is improved based on the SwinTransformer (Swin converter) architecture and combines the Swin Transformer structure and the U-Net structure.

[0124] The SwinUNETR structure uses the Swin Transformer structure as the encoder part, which has strong modeling ability and adaptability. Since the situation inside the blood vessel is relatively complex, such as collateral blood vessels, calcified plaques, aortic dissection, etc., it will cause the internal contour structure of the blood vessel to be incomplete or distorted. In addition, there may be multiple calcified plaques that are discontinuous in the cross-section inside the blood vessel, increasing the difficulty of image segmentation. The SwinUNETR structure draws on the two-dimensional self-attention mechanism in the image classification task. For the situation where there are multiple scattered calcified plaques and incomplete contours in the blood vessel, it can better capture discriminative features and effectively improve the accuracy of the image segmentation result. At the same time, the IVUS image scale is usually large, and downsampling is likely to cause it to lose a large amount of details, resulting in more difficult segmentation of the blood vessel intima. The SwinUNETR structure introduces layer-by-layer partitioning and relative position encoding to process large-scale images, which can maintain a high computational efficiency for the segmentation of IVUS image frames without reducing the segmentation accuracy by downsampling the image frames.

[0125] In addition, the SwinUNETR structure uses the decoder structure of U-Net to generate the image segmentation results. Through this combination, the SwinUNETR structure can better capture the global and local information of IVUS image frames, and still achieve excellent segmentation performance when the IVUS image frames are relatively complex.

[0126] After the sample encoded image is input into the initial segmentation model, the predicted segmentation image obtained includes the intima prediction region and the calcification prediction region.

[0127] In some embodiments, based on the intima prediction region and the intima annotation region, the loss function can be calculated to obtain the first loss. Based on the calcification prediction region and the calcification annotation region, the loss function can be calculated to obtain the second loss. The first loss and the second loss are weighted and summed to obtain the model loss, so as to update the model parameters of the initial segmentation model based on the model loss until the model loss meets the preset stop condition, and an image segmentation model is obtained. Among them, the first loss, the second loss, and the model loss are all loss values obtained by calculating the loss function.

[0128] Since there may be tissues such as collateral vessels and pericardium in multiple image frames of the target blood vessel, when the image segmentation model segments the intima and adventitia of the target blood vessel, it is very easy to segment other tissues, such as Figure 5 As shown in the schematic diagram of the image frame of the target blood vessel in an embodiment. Among them, the area within the circle is the collateral vessel in the target blood vessel.

[0129] In other embodiments, in order to further improve the ability of the image segmentation model to segment the blood vessel intima and calcified plaques and reduce the probability of misidentification, a discriminant model is introduced for adversarial learning. By judging the authenticity of the intima prediction region and the calcification prediction region, the segmentation accuracy of the image segmentation model is improved.

[0130] Such as Figure 6 As shown in the schematic diagram of the training process of the image segmentation model in an embodiment. Among them, the initial discriminant model is a deep learning model for classification. The predicted segmentation image corresponding to the current sample image frame is input into the initial discriminant model, and the first probability obtained is used to represent the probability that the predicted segmentation image is a generated image. The annotated segmentation image can also be used as a training sample of the initial discriminant model. The annotated segmentation image corresponding to the current sample image frame is input into the initial discriminant model, and the second probability obtained is used to represent the probability that the annotated segmentation image is a generated image.

[0131] The model loss includes the loss corresponding to the initial segmentation model and the loss of the initial discriminant model, and can be calculated by substituting the annotated segmentation image, the predicted segmentation image, the first probability, and the second probability into the corresponding loss function.

[0132] The initial segmentation model and the initial discrimination model are trained simultaneously. When the model loss converges to a preset value, the training is stopped to obtain the image segmentation model.

[0133] In this embodiment, by introducing the initial discrimination model to judge the authenticity of the prediction result of the initial segmentation model, the prediction probability of whether the prediction result is a generated image is obtained. The model loss is calculated from the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to multiple sample image frames, so as to train the initial segmentation model and the initial discrimination model simultaneously to obtain the image segmentation model. This method of adversarial learning between the initial segmentation model and the initial discrimination model is beneficial for the segmentation network to generate an image segmentation result close to the manually labeled one for the input image frame. This image segmentation result has high accuracy, which is sufficient to mislead the discrimination model and make it unable to distinguish whether the image segmentation result is an image generated by the model or a truly labeled image.

[0134] In an exemplary embodiment, calculating the model loss based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to multiple sample image frames includes: for each sample image frame, determining the segmentation model loss corresponding to the current sample image frame according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame; determining the first discrimination loss corresponding to the current sample image frame according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame, determining the second discrimination loss corresponding to the current sample image frame according to the first probability and the second probability corresponding to the current sample image frame, and determining the discrimination model loss corresponding to the current sample image frame according to the first discrimination loss and the second discrimination loss corresponding to the current sample image frame; calculating the model loss according to the segmentation model losses and discrimination model losses corresponding to multiple sample image frames.

[0135] Among them, the segmentation model loss is the loss during the training process of the initial segmentation model. In some embodiments, the segmentation model loss can be calculated using the following formula:

[0136]

[0137] Among them, L G represents the segmentation model loss, M represents the predicted segmentation image, represents the labeled segmentation image.

[0138] The discrimination model loss is the loss during the training process of the initial discrimination model. The discrimination model loss includes the first discrimination loss and the second discrimination loss.

[0139] In some embodiments, the discrimination model loss can be calculated using the following formula:

[0140]

[0141]

[0142] L D = λL D1 +(1 - λ)L D2

[0143] where L D1 represents the first discriminant loss, L D2 represents the second discriminant loss, L D represents the discriminant model loss, D(M) represents the first probability, represents the second probability, and λ represents the weight of the first discriminant loss.

[0144] The segmentation model loss, the first discriminant loss, the second discriminant loss, and the model loss are all loss values obtained by calculating the corresponding loss functions. The weighted average of the segmentation model loss and the discriminant model loss corresponding to each of multiple sample image frames is calculated to obtain the model loss.

[0145] In this embodiment, by using the corresponding loss functions for the labeled segmentation image and the predicted segmentation image, the segmentation model loss and the first discriminant loss are respectively calculated. Through the first probability and the second probability, the second discriminant loss is calculated. The discriminant model loss includes the first discriminant loss and the second discriminant loss. The model loss is obtained by the weighted average of the segmentation model loss and the discriminant model loss corresponding to each of multiple sample image frames. Training the initial segmentation model and the initial discriminant model simultaneously is beneficial to improving the segmentation accuracy of the segmentation model.

[0146] To illustrate the method and effect of determining the degree of vascular calcification in this solution in detail, the following is described with a most detailed embodiment:

[0147] As Figure 7 shown is a schematic flowchart of the method for determining the degree of vascular calcification in an embodiment. The method for determining the degree of vascular calcification mainly includes the following three functional parts:

[0148] Data collection and preprocessing: Collect a sequence of IVUS image frames with calcified plaques and perform preprocessing to divide the image frame sequence into a training set, a validation set, and a test set;

[0149] Model construction and training: Based on the above dataset, an image segmentation model and an adventitia segmentation model are respectively constructed and trained based on the training method of generative adversarial learning;

[0150] Calculation of calcification score: The IVUS image frame sequence is respectively input into the image segmentation model and the adventitia segmentation model to generate the intima segmentation region, the adventitia segmentation region, and the segmentation mask of the calcification segmentation region. Three-dimensional reconstruction and measurement are performed based on the masks corresponding to multiple image frames in the image frame sequence, and combined with a preset evaluation strategy, the degree of vascular calcification of the IVUS image frame sequence is evaluated.

[0151] 1. Data collection and preprocessing:

[0152] An IVUS image frame sequence with calcified plaques is collected through an IVUS device. To make the model more robust through training, the collected data comes from multiple subjects in multiple centers under the same instrument. The collected IVUS sample image frame sequence is manually annotated to mark the edges of the calcified plaques, the vascular intima, and the vascular adventitia, and the coordinates of the annotated regions are recorded to obtain the masks for training and testing the segmentation model. Since there may be multiple separate plaques in a single sample image frame, if there are multiple separate calcified plaques in a single IVUS sample image frame, the edges of all the calcified plaques need to be annotated. In some embodiments, multiple sample image frames can be randomly divided into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0153] To improve the recognition accuracy of the model for the vascular contour and the edges of calcified plaques, the sample image frames are subjected to image optimization processing, including gamma correction and non-uniform filtering processing, to suppress image noise and retain image edges.

[0154] To further improve the richness of the sample image frames and enhance the generalization ability of the model, the optimized images can also be subjected to regional occlusion in the embodiments of the present application.

[0155] 2. Model construction and training

[0156] Since the adventitia in the IVUS image frame is more obvious than the intima, and small calcified plaques are difficult to be recognized in some frames, the model construction in the embodiments of the present application is divided into two branches. One branch segments the adventitia, and the other branch segments the intima and calcified plaques. Among them, for the intima and calcified plaque branch, the sample image frames need to be encoded to fuse the temporal features first, and for the adventitia branch, the sample image frames need to be enhanced, and the enhanced image frames are input. The segmentation backbone networks of the two branches can be the same. The training methods of the two branch models can both adopt the method based on generative adversarial learning to improve the similarity between the generated mask and the manually annotated mask during training.

[0157] 2.1 Image encoding based on temporal features

[0158] Due to the presence of blood spots in the target blood vessel, it is difficult to observe the edge of the vascular intima from static IVUS image frames. However, in IVUS image frames, the texture of the vascular adventitia is relatively static, and the texture of the blood in the blood vessel mostly shows scattered dots and continuously changes disorderly. Therefore, by combining the current image frame and the frames before and after the current image frame, according to the texture change differences in different regions of the blood vessel, the blood vessel edge can be captured. In addition, calcified plaques are three-dimensional lesion structures in the blood vessel. They may be small in some frames and difficult to be recognized. With the guidance of the frames before and after, the visual residual when observing IVUS image frames by human vision can be simulated to encode the current image frame, which is beneficial to improving the segmentation accuracy of the vascular intima and calcified plaques.

[0159] After encoding processing, the original single-channel image is processed into a three-channel image. In some embodiments, the sharpening process is represented by S(I), and the current image frame is represented by I n represents, I n-1 、I n-2 respectively represent the previous image frame of the current image frame, I n+1 、I n+2 respectively represent the previous image frame of the current image frame and the subsequent image frame of the current image frame, then the first encoded image can be represented by the following formula:

[0160] The second encoded image is the current image frame. In some embodiments, the second encoded image is represented by can be expressed as:

[0161] Norm [0,255] represents the gray value normalized to between 0 and 255, then the third encoded image can be represented by the following formula:

[0162]

[0163] 2.2 Model Structure

[0164] The image segmentation model and the adventitia segmentation model can be the same or different model structures. Taking the same model structure as an example for illustration. Both branches adopt the SwinUNETR network. As Figure 8 shown is a schematic diagram of the model structure of the image segmentation model and the adventitia segmentation model in an embodiment.

[0165] Due to the organizational structure characteristics of IVUS image frames, the vascular intima will not extend beyond the area where the vascular adventitia is located, and calcified plaques exist between the vascular intima and the vascular adventitia. Therefore, after each mask is output, taking the vascular adventitia segmentation area as the standard, the intima segmentation area and the calcification segmentation area are restricted and corrected. If the intima segmentation area and the calcification segmentation area exceed the adventitia segmentation area, the areas where they intersect with the adventitia segmentation area are respectively taken as the correction areas.

[0166] 2.3 Model Training

[0167] In order to further improve the ability of the image segmentation model to segment the vascular intima and calcified plaques and reduce the probability of misidentification, the embodiment of this application adopts a training method of generative adversarial learning to make the image segmentation result closer to the result manually annotated by experts.

[0168] In some embodiments, the following formula can be used to calculate the segmentation model loss:

[0169]

[0170] Among them, L G represents the segmentation model loss, M represents the predicted segmentation image, represents the annotated segmentation image.

[0171] In some embodiments, the following formula can be used to calculate the discriminant model loss:

[0172]

[0173]

[0174] L D = λL p1 +(1 - λ)L D2

[0175] Among them, L D1 represents the first discriminant loss, L D2 represents the second discriminant loss, L D represents the discriminant model loss, D(M) represents the first probability, represents the second probability, and λ represents the weight of the first discriminant loss.

[0176] The weighted average of the segmentation model loss and the discriminant model loss corresponding to each of the multiple sample image frames is calculated to obtain the model loss. According to the model loss, the model parameters of the initial segmentation model and the initial discriminant model are updated until the model loss meets the preset stop condition, and an image segmentation model is obtained.

[0177] 3. Calcification Scoring

[0178] Use an image segmentation model to segment multiple image frames of a target blood vessel, obtaining image segmentation results corresponding to each of the multiple image frames. Each image segmentation result includes an intima segmentation region and a calcification segmentation region. Use the adventitia segmentation region to separately segment the multiple image frames of the target blood vessel, obtaining adventitia segmentation results corresponding to each of the multiple image frames. Each adventitia segmentation result includes an adventitia segmentation region. Take the edges of each region respectively, and perform alignment and stitching based on the order of each frame in the IVUS image sequence to reconstruct a three-dimensional model of the blood vessel, lumen, and calcified plaque.

[0179] Measure the calcified plaque in the three-dimensional model. The measurement parameters are mainly the maximum length and maximum angle of the calcification.

[0180] The processed IVUS image sequence can be segmented for the intima, external elastic membrane, and calcified plaque. After obtaining the masks of the three tissues to be measured, take the edges of each and perform alignment and stitching based on the order of each frame in the IVUS image sequence to reconstruct a three-dimensional structure of the blood vessel, lumen, and calcified plaque.

[0181] After obtaining the three-dimensional structure of the blood vessel and the calcified plaque, the calcified plaque can be measured. The shape parameters include the angle of the calcification region and the length of the calcification region. Based on the shape parameters and a preset evaluation strategy, determine the degree of blood vessel calcification. Through a preset evaluation strategy table, it can be evaluated whether the input IVUS image frame sequence needs further examination.

[0182] The solution of the embodiment of this application can assist doctors in achieving end-to-end assessment of the in-vessel calcification condition, reduce the time required for diagnosis, improve the efficiency of plaque detection, provide more perspectives for observing the calcified plaque, and reduce the burden on doctors. It adopts an image coding method based on temporal features. By extracting the temporal features of the IVUS images, the contrast of blood, blood vessels, and calcified plaques in the vascular intima can be improved, which is beneficial to enhancing the segmentation performance of the subsequent image segmentation network. Furthermore, it adopts a SwinUNETR segmentation network, which can better capture the global and local information of the IVUS image frames for the complex conditions and a large amount of noise within the IVUS image frames, achieving excellent segmentation performance. The model is trained using a training method of generative adversarial learning, which can further improve the model performance. After segmenting multiple frame images of the IVUS image frame sequence, a three-dimensional model of the vascular intima, adventitia, and calcified plaque is reconstructed three-dimensionally. Combining the shape parameters of the calcified plaque and the preset evaluation strategy can provide a more comprehensive automated method for calcification assessment to assist doctors in accurately diagnosing and treating the degree of calcification of the target blood vessel.

[0183] The above method for determining the degree of vascular calcification obtains an image frame sequence including multiple image frames by scanning a target blood vessel. For each image frame, the current image frame is encoded based on multiple previous image frames and multiple subsequent image frames of the current image frame to obtain an encoded image corresponding to the current image frame. An image segmentation model is used to segment the encoded image corresponding to the current image frame to obtain an image segmentation result corresponding to the current image frame. Each image segmentation result includes an intimal segmentation region and a calcification segmentation region. Since the textures of different regions in the blood vessel have certain differences, encoding based on multiple previous and subsequent image frames integrates the temporal features of multiple image frames in the encoded image, which is beneficial to improving the contrast between the intima and calcification plaques in the blood vessel. Using an image segmentation model to segment the encoded image is beneficial to improving the accuracy of the image segmentation result. Based on the intimal segmentation region and the calcification segmentation region in the image segmentation result, it is beneficial to improve the accuracy of the determined degree of vascular calcification.

[0184] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0185] Based on the same inventive concept, an embodiment of the present application further provides a device for determining the degree of vascular calcification for implementing the above-mentioned method for determining the degree of vascular calcification. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the device for determining the degree of vascular calcification provided below can refer to the limitations on the method for determining the degree of vascular calcification in the above text, and will not be repeated here.

[0186] In an exemplary embodiment, as Figure 9 shown, a device 100 for determining the degree of vascular calcification is provided, including: an acquisition module 120, a segmentation module 140, and a determination module 160, where:

[0187] The acquisition module 120 is configured to acquire an image frame sequence including multiple image frames;

[0188] A segmentation module 140 is configured to segment multiple image frames respectively by using an image segmentation model to obtain image segmentation results corresponding to the multiple image frames respectively. Each image segmentation result includes an intima segmentation region and a calcification segmentation region. The image segmentation model is trained based on sample encoded images corresponding to multiple sample image frames. Each sample encoded image is obtained by encoding a current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame.

[0189] A determination module 160 is configured to determine the degree of vascular calcification according to the intima segmentation regions and the calcification segmentation regions corresponding to the multiple image frames respectively.

[0190] The above-mentioned device for determining the degree of vascular calcification obtains an image frame sequence including multiple image frames, segments the multiple image frames respectively by using an image segmentation model to obtain image segmentation results corresponding to the multiple image frames respectively. Each image segmentation result includes an intima segmentation region and a calcification segmentation region. The image segmentation model is trained based on sample encoded images corresponding to multiple sample image frames. Each sample encoded image is obtained by encoding a current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame. Since the textures of different regions in the blood vessel have certain differences, encoding based on multiple previous and subsequent sample image frames, the encoded image incorporates the temporal features of multiple sample image frames, which is beneficial to improving the contrast between the intima and calcified plaques in the blood vessel. Segmenting the multiple image frames respectively by using the image segmentation model is beneficial to improving the accuracy of the image segmentation result. Determining the degree of vascular calcification based on the intima segmentation region and the calcification segmentation region in the image segmentation result is beneficial to improving the accuracy of the determined degree of vascular calcification.

[0191] In one embodiment, an image segmentation model is used to separately segment the multiple image frames to obtain image segmentation results corresponding to the respective multiple image frames. The segmentation module 140 is further configured to: for each image frame, perform sharpening processing on the multiple previous image frames, the multiple subsequent image frames, and the current image frame respectively, sum the pixel values of the pixel points at the same positions in the obtained multiple sharpening processing results to obtain a first encoded image corresponding to the current image frame; use the current image frame as the second encoded image corresponding to the current image frame; average the pixel values of the pixel points at the same positions in the multiple previous image frames, the multiple subsequent image frames, and the current image frame to obtain a first intermediate image; subtract the pixel values of the pixel points at the same positions in the first intermediate image and the current image frame to obtain a second intermediate image; perform normalization processing on the second intermediate image to obtain a third encoded image corresponding to the current image frame; use the first encoded image, the second encoded image, and the third encoded image corresponding to the current image frame as the encoded image corresponding to the current image frame; use the image segmentation model to separately segment the encoded images corresponding to the respective multiple image frames to obtain image segmentation results corresponding to the respective multiple image frames.

[0192] In one embodiment, the degree of vascular calcification is determined according to the intima segmentation regions and the calcification segmentation regions corresponding to the respective multiple image frames. The determination module 160 is further configured to: reconstruct a three-dimensional model of the target blood vessel according to the intima segmentation regions and the calcification segmentation regions corresponding to the respective multiple image frames; determine the calcified regions in the three-dimensional model and determine the shape parameters of the calcified regions; determine the degree of vascular calcification based on the shape parameters and a preset evaluation strategy.

[0193] In one embodiment, a three-dimensional model of the target blood vessel is reconstructed according to the intima segmentation regions and the calcification segmentation regions corresponding to the respective multiple image frames. The determination module 160 is further configured to: for each image frame, use an adventitia segmentation model to segment the current image frame to obtain an adventitia segmentation result corresponding to the current image frame, where the adventitia segmentation result includes an adventitia segmentation region; respectively correct the intima segmentation region and the calcification segmentation region corresponding to the current image frame according to the adventitia segmentation region corresponding to the current image frame to obtain an intima correction region and a calcification correction region corresponding to the current image frame; determine the scanning order of the multiple image frames, and based on the scanning order, perform three-dimensional reconstruction on the adventitia segmentation regions, the intima correction regions, and the calcification correction regions corresponding to the respective multiple image frames to obtain a three-dimensional model of the target blood vessel.

[0194] In one embodiment, according to the outer membrane segmentation region corresponding to the current image frame, the inner membrane segmentation region and the calcification segmentation region corresponding to the current image frame are respectively corrected to obtain the inner membrane correction region and the calcification correction region corresponding to the current image frame. The determination module 160 is further configured to: use the part of the inner membrane segmentation region corresponding to the current image frame that is within the outer membrane segmentation region corresponding to the current image frame as the inner membrane correction region corresponding to the current image frame; use the part of the calcification segmentation region corresponding to the current image frame that is within the outer membrane segmentation region corresponding to the current image frame as the calcification correction region corresponding to the current image frame.

[0195] In one embodiment, the shape parameters include the calcification region angle and the calcification region length; based on the shape parameters and a preset evaluation strategy, the degree of vascular calcification is determined. The determination module 160 is further configured to: determine the calcification angle score according to the calcification region angle and a preset angle; determine the calcification length score according to the calcification region length and a preset length; perform a weighted sum of the calcification angle score and the calcification length score to obtain a summation result; query a preset mapping relationship to determine the degree of vascular calcification corresponding to the summation result.

[0196] In one embodiment, in terms of the training of the image segmentation model, the vascular calcification degree determination device 100 further includes a training module. The training module is configured to: obtain a plurality of sample image frames obtained by scanning a sample blood vessel and the corresponding labeled segmentation images of the plurality of sample image frames. Each labeled segmentation image includes an inner membrane labeled region and a calcification labeled region; for each sample image frame, encode the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames of the current sample image frame to obtain a sample encoded image corresponding to the current sample image frame; input the sample encoded image corresponding to the current sample image frame into an initial segmentation model to obtain a predicted segmentation image corresponding to the current sample image frame. The predicted segmentation image includes an inner membrane prediction region and a calcification prediction region; input the predicted segmentation image corresponding to the current sample image frame into an initial discrimination model to obtain a first probability of whether the predicted segmentation image is a generated image, and input the labeled segmentation image corresponding to the current sample image frame into the initial discrimination model to obtain a second probability of whether the labeled segmentation image is a generated image. Based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to the plurality of sample image frames, calculate a model loss; update the model parameters of the initial segmentation model and the initial discrimination model according to the model loss until the model loss meets a preset stop condition to obtain an image segmentation model.

[0197] In one embodiment, based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities corresponding to multiple sample image frames, a model loss is calculated. The training module is further configured to: for each sample image frame, determine the segmentation model loss corresponding to the current sample image frame according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame; determine the first discrimination loss corresponding to the current sample image frame according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame, determine the second discrimination loss corresponding to the current sample image frame according to the first probability corresponding to the current sample image frame and the second probability corresponding to the current sample image frame, and determine the discrimination model loss corresponding to the current sample image frame according to the first discrimination loss corresponding to the current sample image frame and the second discrimination loss corresponding to the current sample image frame; calculate the model loss according to the segmentation model losses and discrimination model losses corresponding to multiple sample image frames respectively.

[0198] Each module in the above device for determining the degree of vascular calcification can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in the processor of the computer device in the form of hardware or be independent of it, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above respective modules.

[0199] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 10As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for determining the degree of vascular calcification. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0200] Those skilled in the art can understand that Figure 10 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0201] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0202] In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0203] In an embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by the processor, the steps in the above method embodiments are implemented.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0205] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0206] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0207] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for determining the degree of vascular calcification, characterized in that, the method includes: obtaining an image frame sequence including a plurality of image frames; using an image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the plurality of image frames respectively, each image segmentation result including an intima segmentation region and a calcification segmentation region; the image segmentation model is trained based on sample encoded images corresponding to a plurality of sample image frames respectively; each sample encoded image is obtained by encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames corresponding to the current sample image frame; determining the degree of vascular calcification according to the intima segmentation regions and calcification segmentation regions corresponding to the plurality of image frames respectively.

2. The method according to claim 1, characterized in that, the step of using an image segmentation model to separately segment the plurality of image frames to obtain image segmentation results corresponding to the plurality of image frames respectively includes: for each image frame, performing sharpening processing on a plurality of previous image frames, a plurality of subsequent image frames and the current image frame respectively, summing the pixel values of the pixel points at the same position in the obtained plurality of sharpened processing results to obtain a first encoded image corresponding to the current image frame; using the current image frame as a second encoded image corresponding to the current image frame; averaging the pixel values of the pixel points at the same position in the plurality of previous image frames, the plurality of subsequent image frames and the current image frame to obtain a first intermediate image; calculating the difference between the pixel values of the pixel points at the same position in the first intermediate image and the current image frame to obtain a second intermediate image; performing normalization processing on the second intermediate image to obtain a third encoded image corresponding to the current image frame; using the first encoded image, the second encoded image and the third encoded image corresponding to the current image frame as the encoded image corresponding to the current image frame; using the image segmentation model to separately segment the encoded images corresponding to the plurality of image frames to obtain image segmentation results corresponding to the plurality of image frames respectively.

3. The method according to claim 1, characterized in that, the step of determining the degree of vascular calcification according to the intima segmentation regions and calcification segmentation regions corresponding to the plurality of image frames respectively includes: reconstructing a three-dimensional model of the target blood vessel according to the intima segmentation regions and calcification segmentation regions corresponding to the plurality of image frames respectively; determining the calcification region in the three-dimensional model and determining the shape parameters of the calcification region; determining the degree of vascular calcification based on the shape parameters and a preset evaluation strategy.

4. The method according to claim 3, characterized in that, the step of reconstructing a three-dimensional model of the target blood vessel according to the intima segmentation regions and calcification segmentation regions corresponding to the plurality of image frames respectively includes: for each image frame, using an adventitia segmentation model to segment the current image frame to obtain an adventitia segmentation result corresponding to the current image frame, the adventitia segmentation result including an adventitia segmentation region; According to the outer membrane segmentation region corresponding to the current image frame, correct the inner membrane segmentation region and the calcification segmentation region corresponding to the current image frame respectively, to obtain the inner membrane correction region and the calcification correction region corresponding to the current image frame; Determine the scanning order of multiple image frames. Based on the scanning order, perform three-dimensional reconstruction on the outer membrane segmentation regions, inner membrane correction regions, and calcification correction regions corresponding to the multiple image frames respectively, to obtain the three-dimensional model of the target blood vessel.

5. The method according to claim 4, wherein, The step of correcting the inner membrane segmentation region and the calcification segmentation region corresponding to the current image frame respectively according to the outer membrane segmentation region corresponding to the current image frame to obtain the inner membrane correction region and the calcification correction region corresponding to the current image frame includes: Taking the part of the inner membrane segmentation region corresponding to the current image frame that is within the outer membrane segmentation region corresponding to the current image frame as the inner membrane correction region corresponding to the current image frame; Taking the part of the calcification segmentation region corresponding to the current image frame that is within the outer membrane segmentation region corresponding to the current image frame as the calcification correction region corresponding to the current image frame.

6. The method according to claim 3, wherein, The shape parameters include the calcification region angle and the calcification region length; the step of determining the degree of blood vessel calcification based on the shape parameters and a preset evaluation strategy includes: Determining a calcification angle score according to the calcification region angle and a preset angle; Determining a calcification length score according to the calcification region length and a preset length; Performing weighted summation on the calcification angle score and the calcification length score to obtain a summation result; Querying a preset mapping relationship to determine the degree of blood vessel calcification corresponding to the summation result.

7. The method according to claim 1, wherein, The training step of the image segmentation model includes: Obtaining a plurality of sample image frames obtained by scanning a sample blood vessel and the labeled segmentation images corresponding to the plurality of sample image frames respectively, each labeled segmentation image including an inner membrane labeled region and a calcification labeled region; For each sample image frame, encoding the current sample image frame according to a plurality of previous sample image frames and a plurality of subsequent sample image frames of the current sample image frame, to obtain a sample encoded image corresponding to the current sample image frame; Inputting the sample encoded image corresponding to the current sample image frame into an initial segmentation model to obtain a predicted segmentation image corresponding to the current sample image frame, the predicted segmentation image including an inner membrane prediction region and a calcification prediction region; Inputting the predicted segmentation image corresponding to the current sample image frame into an initial discriminant model to obtain a first probability of whether the predicted segmentation image is a generated image, and inputting the labeled segmentation image corresponding to the current sample image frame into the initial discriminant model to obtain a second probability of whether the labeled segmentation image is a generated image, Calculating a model loss based on the labeled segmentation images, predicted segmentation images, the first probability, and the second probability corresponding to the plurality of sample image frames respectively; Update the model parameters of the initial segmentation model and the initial discriminant model according to the model loss until the model loss meets a preset stopping condition, and obtain an image segmentation model.

8. The method according to claim 7, wherein, calculating the model loss based on the labeled segmentation images, predicted segmentation images, first probabilities, and second probabilities respectively corresponding to multiple sample image frames includes: For each sample image frame, determine the segmentation model loss corresponding to the current sample image frame according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame; Determine the first discriminant loss corresponding to the current sample image frame according to the labeled segmentation image corresponding to the current sample image frame and the predicted segmentation image corresponding to the current sample image frame, determine the second discriminant loss corresponding to the current sample image frame according to the first probability corresponding to the current sample image frame and the second probability corresponding to the current sample image frame, and determine the discriminant model loss corresponding to the current sample image frame according to the first discriminant loss corresponding to the current sample image frame and the second discriminant loss corresponding to the current sample image frame; Calculate the model loss according to the segmentation model losses and discriminant model losses respectively corresponding to multiple sample image frames.

9. An apparatus for determining the degree of vascular calcification, wherein, the apparatus includes: an acquisition module, configured to acquire an image frame sequence including multiple image frames; a segmentation module, configured to separately segment the multiple image frames by using an image segmentation model to obtain image segmentation results respectively corresponding to the multiple image frames, and each image segmentation result includes an intima segmentation region and a calcification segmentation region; the image segmentation model is trained based on sample encoded images respectively corresponding to multiple sample image frames; each sample encoded image is obtained by encoding the current sample image frame according to multiple previous sample image frames and multiple subsequent sample image frames corresponding to the current sample image frame; a determination module, configured to determine the degree of vascular calcification according to the intima segmentation regions and calcification segmentation regions respectively corresponding to multiple image frames.

10. A computer device, including a memory and a processor, where the memory stores a computer program, wherein, when the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

11. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images