Blood vessel recognition model training method and device, blood vessel recognition method and device, electronic equipment, storage medium and computer program product

Abstract

The invention provides a training method of a blood vessel recognition model, a blood vessel recognition method and device, electronic equipment, a storage medium and a computer program product. The training method comprises the steps of inputting each image group into a corresponding MSFUnet network; inputting a plurality of first blood vessel recognition results corresponding to the plurality of MSFUnet networks into an LSTM network; calculating a first loss based on the first blood vessel recognition result and the first image tag output by each MSFUnet network; calculating a second loss based on a blood vessel recognition final result output by the LSTM network and the second image tag; and adjusting a parameter of the MSFUnet network based on the first loss, and adjusting a parameter of the LSTM network based on the second loss. By using the multi-frame fusion algorithm L-MSFUnet, that is, by using a plurality of angiography images to carry out blood vessel recognition, the considered data volume is more, the data content is richer, the generation of the aorta abdominalis blood vessel tree with more comprehensive coverage can be ensured, and the accuracy of blood vessel recognition can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for a blood vessel recognition model, a blood vessel recognition method, a device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] In interventional surgery, it is crucial to accurately identify the aortic vessels and their branch vessels from Digital Subtraction Angiography (DSA) images. In endovascular repair of abdominal aortic aneurysm, in order to visualize the blood vessels, doctors often need to inject contrast agents into the blood vessels multiple times to display different vascular locations, thereby guiding the selection and placement of the implanted stent. In addition, since the blood is constantly flowing, the abdomen of the organism needs to be exposed to X-rays for a long time, which will obtain a frame sequence containing multiple DSA images.

[0003] In the related art, when using DSA image sequences for vascular identification, the most representative image frame is selected from the DSA image sequence to determine the shape and position of the blood vessels. However, due to the large abdominal space, it is difficult to obtain a complete abdominal aortic vascular tree in a single image frame, which may lead to inaccurate vascular identification and cause serious medical accidents. Summary of the invention

[0004] The present disclosure provides a training method for a vascular recognition model, a vascular recognition method, a device, an electronic device, a storage medium, and a computer program product, to at least solve the problem in the related art that it is difficult to obtain a complete abdominal aortic vascular tree in a single image frame, which may lead to inaccurate vascular recognition and thus cause serious medical accidents. The technical solution of the present disclosure is as follows: According to a first aspect of an embodiment of the present disclosure, a training method for a blood vessel recognition model is provided, wherein the blood vessel recognition model includes an LSTM network and multiple MSFUnet networks, and the training method includes: acquiring multiple blood vessel images, wherein the multiple blood vessel images are images obtained by injecting a contrast agent into a blood vessel of a biological body; dividing the multiple blood vessel images into multiple image groups, wherein the multiple image groups correspond one-to-one to the multiple MSFUnet networks, and each image group includes at least two blood vessel images; inputting each image group into the MSFUnet network corresponding to the image group to obtain a first blood vessel recognition result output by the MSFUnet network; inputting multiple first blood vessel recognition results corresponding one-to-one to the multiple MSFUnet networks into the LSTM network to obtain a blood vessel recognition result; Identify the final result; calculate a first loss based on the first blood vessel recognition result output by each MSFUnet network and the first image label bound to the image group corresponding to the MSFUnet network, wherein the first image label is an image label containing blood vessel morphology marked based on at least two of the blood vessel images contained in the image group; calculate a second loss based on the final result of blood vessel recognition output by the LSTM network and the second image label bound to the multiple blood vessel images, wherein the second image label is an image label containing blood vessel morphology marked based on the multiple blood vessel images; train the blood vessel recognition model by adjusting the parameters of the MSFUnet network based on the first loss and adjusting the parameters of the LSTM network based on the second loss.

[0005] Optionally, the MSFUnet network includes a dual attention module, a channel attention module, multiple encoder modules, multiple residual convolution modules and multiple optical flow modules; the step of inputting each image group into the MSFUnet network corresponding to the image group to obtain a first blood vessel recognition result output by the MSFUnet network includes: encoding the image group through the multiple encoder modules to obtain an encoding result; inputting the encoding result into the dual attention module to obtain a first attention result; performing optical flow processing on the first attention result through the multiple optical flow modules to obtain an optical flow result; for the first residual convolution module among the multiple residual convolution modules, based on the first attention result, the output of the encoder module at the same layer as the residual convolution module, and the output of the optical flow module at the same layer as the residual convolution module, Obtain a residual convolution result of the residual convolution module; for each residual convolution module from the second to the nth of the multiple residual convolution modules, based on the output of the residual convolution module of the previous layer of the residual convolution module, the output of the encoder module of the same layer as the residual convolution module, and the output of the optical flow module of the same layer as the residual convolution module, obtain the residual convolution result of the residual convolution module, wherein n is an integer greater than or equal to 2; obtain a second attention result through the channel attention module based on the first attention result and the residual convolution result of each residual convolution module in the multiple residual convolution modules; obtain a first blood vessel recognition result of the MSFUnet network based on the residual convolution result output by the residual convolution module located in the last layer of the multiple residual convolution modules and the second attention result.

[0006] Optionally, the training method also includes: for each MSFUnet network, calculating a third loss based on the optical flow results output by an optical flow module located in the last layer of multiple optical flow modules included in the MSFUnet network and the first image label; adjusting the parameters of the MSFUnet network based on the first loss includes: adjusting the parameters of the MSFUnet network based on both the first loss and the third loss.

[0007] Optionally, each encoder module includes a downsampling module; encoding the image group through the multiple encoder modules to obtain the encoding result includes: for the first encoder module, obtaining the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch of the image group through the downsampling module included in the encoder module, and splicing the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch to obtain an output result; for each encoder module except the first encoder module among the multiple encoder modules, obtaining the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch of the output result of the previous encoder module through the downsampling module included in the encoder module, and splicing the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch to obtain an output result; the output result of the last encoder module among the multiple encoder modules is used as The encoding result; wherein the down-sampling feature of the average pooling branch is obtained in the following manner: performing an average pooling operation on the image group or the output result of the previous encoder module to obtain an average pooling result; sequentially performing a convolution operation, a pixel rearrangement operation, a Softmax operation, and a Hamming window dot multiplication operation on the average pooling result to obtain a first convolution kernel; convolving the first convolution kernel with the average pooling result to obtain the down-sampling feature of the average pooling branch; the down-sampling feature of the maximum pooling branch is obtained in the following manner: performing a maximum pooling operation on the image group or the output result of the previous encoder module to obtain a maximum pooling result; sequentially performing a convolution operation, a pixel rearrangement operation, a Softmax operation, and a Hamming window dot multiplication operation on the maximum pooling result to obtain a second convolution kernel; convolving the second convolution kernel with the maximum pooling result to obtain the down-sampling feature of the maximum pooling branch.

[0008] According to a second aspect of an embodiment of the present disclosure, a blood vessel recognition method is provided. The blood vessel recognition method is implemented based on a blood vessel recognition model trained according to a training method of the present disclosure, and the blood vessel recognition method includes: obtaining a plurality of target blood vessel images to be identified, wherein the plurality of target blood vessel images to be identified are images obtained by injecting a contrast agent into a blood vessel of a biological body; dividing the plurality of target blood vessel images to be identified into a plurality of image groups to be identified, wherein the plurality of image groups to be identified correspond one-to-one to the plurality of MSFUnet networks, and each image group to be identified includes at least two target blood vessel images to be identified; inputting each image group to be identified into an MSFUnet network corresponding to the image group to be identified, and obtaining a first blood vessel recognition target result output by the MSFUnet network; and inputting a plurality of first blood vessel recognition target results corresponding one-to-one to the plurality of MSFUnet networks into the LSTM network, and obtaining a final target result of blood vessel recognition.

[0009] According to a third aspect of an embodiment of the present disclosure, a training device for a blood vessel recognition model is provided, wherein the blood vessel recognition model includes an LSTM network and multiple MSFUnet networks, and the training device includes: a blood vessel image acquisition module, configured to acquire multiple blood vessel images, wherein the multiple blood vessel images are images obtained by injecting a contrast agent into a blood vessel of a biological body; an image group division module, configured to divide the multiple blood vessel images into multiple image groups, wherein the multiple image groups correspond one-to-one to the multiple MSFUnet networks, and each image group includes at least two of the blood vessel images; a first blood vessel recognition result acquisition module, configured to input each image group into the MSFUnet network corresponding to the image group to obtain a first blood vessel recognition result output by the MSFUnet network; and a blood vessel recognition final result acquisition module, configured to input multiple first blood vessel recognition results corresponding one-to-one to the multiple MSFUnet networks into the MSFUnet network. The LSTM network is used to obtain the final result of blood vessel recognition; a first loss calculation module is configured to calculate a first loss based on the first blood vessel recognition result output by each MSFUnet network and the first image label bound to the image group corresponding to the MSFUnet network, wherein the first image label is an image label containing blood vessel morphology annotated based on at least two of the blood vessel images contained in the image group; a second loss calculation module is configured to calculate a second loss based on the final result of blood vessel recognition output by the LSTM network and the second image label bound to the multiple blood vessel images, wherein the second image label is an image label containing blood vessel morphology annotated based on the multiple blood vessel images; a training module is configured to train the blood vessel recognition model by adjusting the parameters of the MSFUnet network based on the first loss and adjusting the parameters of the LSTM network based on the second loss.

[0010] Optionally, the MSFUnet network includes a dual attention module, a channel attention module, multiple encoder modules, multiple residual convolution modules and multiple optical flow modules; the first blood vessel recognition result acquisition module is configured to: encode the image group through the multiple encoder modules to obtain an encoding result; input the encoding result into the dual attention module to obtain a first attention result; perform optical flow processing on the first attention result through the multiple optical flow modules to obtain an optical flow result; for the first residual convolution module among the multiple residual convolution modules, based on the first attention result, the output of the encoder module at the same layer as the residual convolution module, and the output of the optical flow module at the same layer as the residual convolution module, obtain the residual convolution result of the residual convolution module; for For each residual convolution module from the second to the nth of the multiple residual convolution modules, based on the output of the residual convolution module of the previous layer of the residual convolution module, the output of the encoder module of the same layer as the residual convolution module, and the output of the optical flow module of the same layer as the residual convolution module, obtain the residual convolution result of the residual convolution module, where n is an integer greater than or equal to 2; obtain a second attention result through the channel attention module based on the first attention result and the residual convolution result of each residual convolution module in the multiple residual convolution modules; obtain a first blood vessel recognition result of the MSFUnet network based on the residual convolution result output by the residual convolution module located in the last layer of the multiple residual convolution modules and the second attention result.

[0011] Optionally, the training device also includes: a third loss calculation module, configured to calculate a third loss for each MSFUnet network based on the optical flow result output by an optical flow module located at the last layer among multiple optical flow modules included in the MSFUnet network and the first image label; the training module is configured to: adjust the parameters of the MSFUnet network based on both the first loss and the third loss.

[0012] Optionally, each encoder module includes a downsampling module; the first blood vessel identification result acquisition module is configured as follows: for the first encoder module, obtaining the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch of the image group through the downsampling module included in the encoder module, and splicing the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch to obtain an output result; for each encoder module except the first encoder module among the multiple encoder modules, obtaining the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch of the output result of the previous encoder module through the downsampling module included in the encoder module, and splicing the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch to obtain an output result; using the output result of the last encoder module among the multiple encoder modules as the encoding result ; Wherein, the down-sampling features of the average pooling branch are obtained in the following manner: performing an average pooling operation on the image group or the output result of the previous encoder module to obtain an average pooling result; sequentially performing a convolution operation, a pixel rearrangement operation, a Softmax operation, and a Hamming window dot multiplication operation on the average pooling result to obtain a first convolution kernel; convolving the first convolution kernel with the average pooling result to obtain the down-sampling features of the average pooling branch; the down-sampling features of the maximum pooling branch are obtained in the following manner: performing a maximum pooling operation on the image group or the output result of the previous encoder module to obtain a maximum pooling result; sequentially performing a convolution operation, a pixel rearrangement operation, a Softmax operation, and a Hamming window dot multiplication operation on the maximum pooling result to obtain a second convolution kernel; convolving the second convolution kernel with the maximum pooling result to obtain the down-sampling features of the maximum pooling branch.

[0013] According to a fourth aspect of an embodiment of the present disclosure, a blood vessel identification device is provided. The blood vessel identification device is implemented based on a blood vessel identification model trained according to the training method of the present disclosure, and the blood vessel identification device includes: a target blood vessel image acquisition module, configured to acquire a plurality of target blood vessel images to be identified, wherein the plurality of target blood vessel images to be identified are images obtained by injecting a contrast agent into a blood vessel of a biological body; a to-be-identified image group division module, configured to divide the plurality of target blood vessel images to be identified into a plurality of to-be-identified image groups, wherein the plurality of to-be-identified image groups correspond one-to-one to the plurality of MSFUnet networks, and each to-be-identified image group includes at least two of the target blood vessel images to be identified; a first blood vessel identification target result acquisition module, configured to input each to-be-identified image group into the MSFUnet network corresponding to the to-be-identified image group, and obtain a first blood vessel identification target result output by the MSFUnet network; and a blood vessel identification final target result acquisition module, configured to input a plurality of first blood vessel identification target results corresponding one-to-one to the plurality of MSFUnet networks into the LSTM network, and obtain a blood vessel identification final target result.

[0014] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a blood vessel recognition model training method or a blood vessel recognition method according to the present disclosure.

[0015] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the training method of a blood vessel recognition model or the blood vessel recognition method according to the present disclosure.

[0016] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the training method of a blood vessel recognition model or the blood vessel recognition method according to the present disclosure is implemented.

[0017] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: In the present disclosure, by using the multi-frame fusion algorithm L-MSFUnet, that is, by using multiple angiography images to perform blood vessel identification, the amount of data considered is larger and the data content is richer, thereby ensuring the generation of a more comprehensive abdominal aortic vascular tree, that is, the accuracy of blood vessel identification can be effectively improved, thereby effectively avoiding the occurrence of medical accidents.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0020] Figure 1 is a flow chart illustrating a method for training a blood vessel recognition model according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic structural diagram showing a blood vessel recognition model according to an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram illustrating intra-frame position coding and time frame coding according to an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram showing the structure of an SFB according to an exemplary embodiment of the present disclosure; Figure 5 is a schematic diagram showing the structure of an optical flow module (Flow) according to an exemplary embodiment of the present disclosure; Figure 6 is a schematic diagram showing the structure of a residual convolution module ResDe according to an exemplary embodiment of the present disclosure; Figure 7 is a schematic diagram showing the structure of a CA module according to an exemplary embodiment of the present disclosure; Figure 8 is a structural schematic diagram showing a downsampling module (FreqPass) according to an exemplary embodiment of the present disclosure; Fig. 9 is a schematic diagram showing a comparison between a single-frame predicted segmentation result of different blood vessel regions and a blood vessel fusion segmentation result obtained from a complete sequence according to an exemplary embodiment of the present disclosure; Fig.10 is a flow chart illustrating a blood vessel identification method according to an exemplary embodiment of the present disclosure; Fig.11 is a block diagram illustrating a training apparatus for a blood vessel recognition model according to an exemplary embodiment of the present disclosure; Fig.12 is a block diagram illustrating a blood vessel identification device according to an exemplary embodiment of the present disclosure; Fig.13 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0023] Abdominal aortic aneurysm is a local permanent dilatation of the abdominal aorta. Endovascular repair of abdominal aortic aneurysm has rapidly developed into the preferred method for treating abdominal aortic aneurysm with low morbidity and mortality. In interventional surgery, it is crucial to accurately identify the aortic vessels and their branch vessels from DSA images. At this time, doctors often need to inject contrast agents into the user's blood vessels multiple times to show different vascular locations, and then determine the selection and placement of the implanted stent. In a complete sequence of vascular image frames, there are often many noises, including intestines, kidneys, complex fine blood vessels, and relatively blurred vascular edges with mural thrombi. In addition, due to the large abdominal space, it is difficult to obtain a complete abdominal aortic vascular tree angiography in one frame of image. Inaccurate vascular identification may lead to incorrect placement of endovascular grafts, leading to surgical failure. In addition, traditional methods usually only focus on the segmentation of the abdominal aorta or local blood vessels, ignoring the criticality of constructing a complete vascular tree including branched vessels.

[0024] In order to solve the above-mentioned problems in the related art, the training method of the vascular recognition model, the vascular recognition method, the device, the electronic device, the storage medium and the computer program product proposed in the present disclosure use the multi-frame fusion algorithm L-MSFUnet, that is, use multiple angiography images to perform vascular recognition, so that the amount of data considered is larger and the data content is richer, thereby ensuring the generation of a more comprehensive abdominal aorta vascular tree, that is, the accuracy of vascular recognition can be effectively improved, thereby effectively avoiding the occurrence of medical accidents.

[0025] Figure 1 The present invention is a flowchart showing a training method of a blood vessel recognition model according to an exemplary embodiment of the present disclosure. The blood vessel recognition model may include a long short-term memory network (LSTM) and a plurality of multi-frame sequence fusion U-net (MSFUnet).

[0026] Reference Figure 1In step 101, a plurality of vascular images may be acquired, wherein the plurality of vascular images may be images acquired by injecting a contrast agent into a blood vessel of a living body.

[0027] In step 102, the plurality of vascular images may be divided into a plurality of image groups, wherein the plurality of image groups may correspond one-to-one to a plurality of MSFUnet networks, and each image group may include at least two vascular images. Figure 2 is a schematic diagram showing the structure of a blood vessel recognition model according to an exemplary embodiment of the present disclosure. Figure 2 The overall structure of the blood vessel recognition model, namely the L-MSFUne model, can be composed of an encoder-decoder segmentation network MSFUnet network and an LSTM network module. For example, Figure 2 3 MSFUnet networks are shown in FIG. 3 , each MSFUnet network may correspond to an image group, and each image group may include 3 vascular images arranged in sequence.

[0028] It should be noted that the input image can also be divided into patches of size 5, and the step length of the division can be 4. Through this overlapping division operation, the edge effect of the image can be reduced. In addition, when encoding, in addition to encoding the position within the frame, the time information of the image can also be encoded according to the input order of the vascular image. For example, Figure 2 The time codes of the three frames of blood vessel images contained in each image group shown in FIG. 1 can be: 0, 1, 2 respectively.

[0029] Figure 3 is a schematic diagram showing intra-frame position coding and time frame coding according to an exemplary embodiment of the present disclosure. Figure 3 , a single image group may include 3 frames of vascular images, and "0, 1, 2, 3" under the first frame of vascular image may indicate the position codes of the 4 image regions included in the frame of vascular image, and the arrangement positions of the 4 image regions in the first frame of vascular image may be, but are not limited to, "from left to right, from top to bottom". Moreover, "0, 0, 0, 0" under the first frame of vascular image may indicate that the time code of the frame of vascular image is 0.

[0030] The “0, 1, 2, 3” under the second frame of vascular image may indicate the position codes of the four image regions included in the frame of vascular image, and the arrangement positions of the four image regions in the second frame of vascular image may be, but are not limited to, “from left to right, from top to bottom”. Furthermore, the “1, 1, 1, 1” under the second frame of vascular image may indicate that the time code of the frame of vascular image is 1.

[0031] The "0, 1, 2, 3" under the third frame of vascular image can represent the position coding of the four image regions contained in the frame of vascular image, and the arrangement positions of the four image regions in the third frame of vascular image can be, but are not limited to, "from left to right, from top to bottom". In addition, the "2, 2, 2, 2" under the third frame of vascular image can represent that the time coding of the frame of vascular image is 2. In this way, by introducing time frame coding, the feature continuity across multiple frames can be enhanced.

[0032] In step 103, each image group may be input into the MSFUnet network corresponding to the image group to obtain the first blood vessel recognition result output by the MSFUnet network. For example, as mentioned above, each image group may include three consecutive frames of DSA images, and these DSA images may be sampled to 512 512 size.

[0033] According to an exemplary embodiment of the present disclosure, the MSFUnet network may include a double-attention module, a channel attention module (Channel Attention, CA), multiple encoder modules (SpatialFrequency Block, SFB), multiple residual convolution modules (resdecoder block, ResDe) and multiple optical flow modules (flow attention).

[0034] Reference Figure 2 , the image group can be encoded by multiple encoder modules to obtain an encoding result. Then, the encoding result can be input into the dual attention module to obtain a first attention result. Next, the first attention result can be processed by multiple optical flow modules to obtain an optical flow result.

[0035] Then, for the first residual convolution module among the multiple residual convolution modules, the residual convolution result of the residual convolution module can be obtained based on the first attention result, the output of the encoder module at the same layer as the residual convolution module, and the output of the optical flow module at the same layer as the residual convolution module; for each residual convolution module from the second to the nth among the multiple residual convolution modules, the residual convolution result of the residual convolution module can be obtained based on the output of the residual convolution module of the previous layer of the residual convolution module, the output of the encoder module at the same layer as the residual convolution module, and the output of the optical flow module at the same layer as the residual convolution module, where n can be an integer greater than or equal to 2.

[0036] Next, the second attention result can be obtained through the channel attention module based on the first attention result and the residual convolution result of each residual convolution module in the multiple residual convolution modules. Then, the first blood vessel recognition result of the MSFUnet network can be obtained based on the residual convolution result output by the residual convolution module located in the last layer of the multiple residual convolution modules and the second attention result. Specifically, the residual convolution result output by the residual convolution module of the last layer can be Kronecker product operation with the second attention result, and then the first blood vessel recognition result of the MSFUnet network can be obtained.

[0037] Figure 4 is a schematic diagram showing the structure of an SFB according to an exemplary embodiment of the present disclosure. Figure 4 , the SFB module can be composed of 2 self-attention modules and a downsampling module (FreqPass). When calculating the self-attention feature, the feature can be divided into 8 The calculation window of 8 patches can be used, and the self-attention feature (WMSA) can be calculated within the calculation window. In addition, in order to improve the context of the entire image, a window shifting operation (SWMSA) can be adopted in the second self-attention module. The self-attention module can adopt a multi-head self-attention mechanism, that is, different numbers can be used between different layers. For example, the first to fifth layers can be set to: 4, 8, 16, 32, 64 respectively.

[0038] It should be noted that, refer to Figure 2 , the decoder included in the MSFUnet network can be composed of three parts, namely: ResDe module, Flow module and CA module. After the feature extraction of five layers of SFB, the extracted features can be input into the double-attention layer, and then the output of the double-attention layer can be used as the input of the optical flow attention module (Flow) and the input of the residual convolution module (ResDe). Exemplarily, the "double-attention layer" can be an SFB without FreqPass.

[0039] Figure 5 is a schematic diagram showing the structure of an optical flow module (Flow) according to an exemplary embodiment of the present disclosure. Figure 5, the optical flow features can be calculated through three adjacent frames of images, where the first frame and the second frame can be used as the first set of optical flow features; the second frame and the third frame can be used as another set of optical flow features. The optical flow features can be expanded in a forward manner and can be used as the input of the cross attention module respectively. The cross attention module can use the window-based multihead cross attention mechanism (Window-based Multihead Cross Attention, WMCA) for feature extraction.

[0040] Specifically, first, a self-attention calculation can be performed on the forward features (for example, Figure 5 The WMSA module in the cross attention module) can be used as the query (Q) in the cross attention module. Then, the key (K) and value (V) of the cross attention can also be calculated using the backward features. Through the above steps, the feature results of the cross attention can be finally calculated.

[0041] Next, the feature results of the cross-attention can be input into the Global Correlation Module for optical flow prediction. The global correlation can be calculated by dot product similarity and normalized using the Softmax function. In addition, in order to ensure the consistency of optical flow feature information between different scales (feature layers) and to promote the propagation accuracy from low-resolution optical flow features to high-resolution optical flow features, the deep optical flow features and the optical flow features of the current layer can be further cross-attention calculated (for example Figure 5 In this way, by setting up a decoding structure that integrates optical flow characteristics, the spatiotemporal continuity of blood vessels in image sequences can be ensured.

[0042] As mentioned above, in the decoder, the input of each layer of ResDe can consist of the output of the previous layer of ResDe and the skip connection between the same layer. When the skip connection is applied, the output from the encoder module SFB of the same layer and the output of the Flow module of the same layer can be fed into ResDe. Figure 6 is a schematic diagram showing the structure of a residual convolution module ResDe according to an exemplary embodiment of the present disclosure. Figure 6 , ResDe can include an upsampling layer and two residual blocks with the same operation, wherein the residual block can include convolution (conv), batch normalization (BN) and linear rectification function (ReLU). Exemplarily, the upsampling layer can use content-aware feature recombination (Carafe) for upsampling. Compared with the traditional bilinear upsampling method, Carafe upsampling has a larger receptive field and can better preserve the semantic information of the feature map.

[0043] It should be noted that after the double-attention module and multiple residual convolution modules ResDe, five output features of different resolutions can be obtained. The resolution of these features can be expanded to the same scale (h / 4×w / 4) while keeping the number of channels unchanged. These five output features of different resolutions can be input into the channel attention module (CA) to obtain attention features with channel weights. Figure 7 is a schematic diagram showing the structure of a CA module according to an exemplary embodiment of the present disclosure. Figure 7 , the CA module can extract information from different channels through global average pooling (GAP) and global maximum pooling (GMP) operations. This mechanism allows the network to learn global information and selectively strengthen important features while suppressing less useful information. In this way, the expressiveness of feature channels can be enhanced through the CA module, thereby improving the representation ability of the network.

[0044] It should be noted that, refer to Figure 2 , the prediction results of multiple MSFUnet networks can be input into the LSTM network module in the order of frames. Furthermore, the prediction results of the MSFUnet network and the optical flow features output by the optical flow module of the last layer of the multiple optical flow modules contained in the MSFUnet network can be simultaneously input into the LSTM network module. And, its size can be: [n, 1, 4, 512, 512], where n can be the number of frames in the sequence. The LSTM network module can be composed of a convolutional layer, 4 LSTM layers and a linear layer.

[0045] Specifically, the convolution layer can be a point convolution operation, the number of input channels can be 4, the number of output channels can be 2, the convolution kernel can be 1, and the step size can be 1. Then, the result of the convolution layer output can be resized. For example, it can be transformed from [n, 1, 2, 512, 512] to [1, n, 512 512 2] and can be sent to the LSTM layer. In addition, the number of hidden layers of the LSTM layer can be set to 4 and the hidden layer size can be set to 1024. Next, the output of the LSTM layer can be passed through a linear layer and reshaped to [1, 2, 512, 512]. In this way, the segmentation results from different frameworks can be further fused through the LSTM network module, and a comprehensive abdominal aorta vascular tree can be generated.

[0046] It should be noted that since the image development effect is affected by the contrast agent perfusion speed, blood vessel shape and development principle, DSA images often have uneven blood vessel development and blurred blood vessel tissue edges. To solve this problem, the FreqPass module with Hamming filtering can be used to improve the uniformity within the blood vessel class and the difference between the edges of blood vessel tissue classes.

[0047] According to an exemplary embodiment of the present disclosure, each encoder module (SFB) may include a down-sampling module (FreqPass). Figure 8 is a structural diagram showing a down-sampling module (FreqPass) according to an exemplary embodiment of the present disclosure.

[0048] For the first encoder module, the downsampling features of the average pooling branch (avg pool) and the downsampling features of the maximum pooling branch (max pool) of the image group can be obtained through the downsampling module contained in the encoder module, and the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch can be concatenated to obtain the output result. Furthermore, the concatenation result of the downsampling features can be operated using pointwise convolution to obtain the encoding result, that is, the final downsampling feature Xdown (size is [b, 2c, h / 2, w / 2]).

[0049] For each encoder module except the first encoder module among the multiple encoder modules, the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch of the output result of the previous encoder module can be obtained through the downsampling module included in the encoder module, and the downsampling features of the average pooling branch and the downsampling features of the maximum pooling branch can be spliced ​​to obtain the output result. In this way, the output result of the last encoder module among the multiple encoder modules can be used as the encoding result for inputting the double-attention module.

[0050] It should be noted that the aforementioned “down-sampling features of the average pooling branch” can be obtained in the following way: For the image group or the output result of the previous encoder module, for example, the input feature X (size is [b,c,h,w]) can be average pooled to obtain the average pooling result. Then, the convolution operation, pixel shuffle operation, and softmax operation can be performed on the average pooling result in sequence to obtain a feature with a size of [b,hw / 4, 3, 3]. Figure Xs. Next, the Hamming window dot multiplication operation can be used to obtain the first convolution kernel Xw (size is [b, 9, h / 2, w / 2]). Then, the first convolution kernel Xw can be convolved with the average pooling result to obtain the down-sampled features of the average pooling branch.

[0051] The aforementioned "down-sampling features of the maximum pooling branch" can be obtained in the following way: For the image group or the output result of the previous encoder module, for example, the input feature X (size is [b,c,h,w]) can be subjected to the max pooling operation to obtain the max pooling result. Then, the convolution operation, pixel shuffle operation, and softmax operation can be performed on the max pooling result in sequence to obtain a feature with size [b,hw / 4, 3, 3]. Figure X s. Next, the Hamming window dot multiplication operation can be used to obtain the second convolution kernel Xw (size is [b, 9, h / 2, w / 2]). Then, the second convolution kernel Xw can be convolved with the maximum pooling result to obtain the down-sampled features of the maximum pooling branch (max pool). In this way, the uniformity of blood vessels in intra-class images and the difference between classes can be effectively improved by using the feature extraction layer with the FreqPass module.

[0052] In step 104, a plurality of first blood vessel recognition results corresponding to a plurality of MSFUnet networks may be input into the LSTM network to obtain a final blood vessel recognition result.

[0053] In step 105, the first loss may be calculated based on the first blood vessel recognition result output by each MSFUnet network and the first image label bound to the image group corresponding to the MSFUnet network. The first image label may be an image label containing blood vessel morphology annotated based on at least two blood vessel images included in the image group, that is, the first image label may be an image label containing blood vessel morphology annotated by a doctor based on at least two blood vessel images included in the image group.

[0054] For example, refer to Figure 2 , a total of 3 MSFUnet networks are shown, taking the first MSFUnet network as an example. The image group corresponding to the first MSFUnet network contains a total of 3 vascular images. Doctors can annotate the vascular morphology based on these 3 vascular images according to their own experience as the first image label bound to the image group corresponding to the first MSFUnet network.

[0055] In step 106, the second loss may be calculated based on the final result of blood vessel recognition output by the LSTM network and a second image label bound to the plurality of blood vessel images, wherein the second image label may be an image label containing blood vessel morphology marked based on the plurality of blood vessel images.

[0056] Specifically, the second loss can be a sequence-level loss (pre_sequence, label_sequence) obtained by calculating the difference between the result of fusing the single-frame prediction image output by the MSFUnet network through the LSTM network module and the sequence label, and the loss can be calculated using Focal Loss. The aforementioned second image label can be an image label containing vascular morphology marked by a doctor based on multiple vascular images, that is, the doctor can mark the vascular morphology based on the multiple vascular images initially input into the vascular recognition model according to his own experience as the second image label.

[0057] In step 107 , the blood vessel recognition model may be trained by adjusting the parameters of the MSFUnet network based on the first loss, and adjusting the parameters of the LSTM network based on the second loss.

[0058] According to an exemplary embodiment of the present disclosure, for each MSFUnet network, the third loss may also be calculated based on the optical flow result output by the optical flow module (flow) of the last layer of the multiple optical flow modules (flow) included in the MSFUnet network and the first image label. Figure 2 The third loss can be calculated based on the optical flow result output by the optical flow module of the last layer of the five optical flow modules included in the MSFUnet network and the first image label. Next, the parameters of the MSFUnet network can be adjusted based on the aforementioned first loss and the third loss.

[0059] It should be noted that the sum of the first loss and the third loss can also be called a frame-level loss, that is, the frame-level loss can be obtained by calculating the first image label and the prediction result (pre_MSF, label_frame) of the MSFUnet model and the prediction result (pre_flow, label_frame) of the optical flow module and adding them. In addition, both (pre_MSF, label_frame) and (pre_flow, label_frame) can be calculated using Focal Loss and weighted Dice Loss.

[0060] Specifically, Focal Loss can be calculated by the following formula:

[0061]

[0062] Where M is the number of categories, is the value of the sample category after one-hot encoding, is the predicted probability of belonging to category c, is 0.25 and γ is 2.

[0063] Weighted Dice Loss, that is It can be calculated by the following formula:

[0064]

[0065]

[0066] Among them, M is the number of categories, n is the number of regions in a single frame, B represents the boundary length of the target, S represents the size of the target, X is the set of predicted values, and Y is the set of true values. In this way, the segmentation accuracy of blood vessels can be significantly improved by using the hybrid loss function.

[0067] It should be noted that the data set used in the present disclosure may include abdominal DSA data of 93 patients, and the collected cases can be randomly divided into 3 groups: data of 60 patients can be assigned to the training set; data of 15 patients can be assigned to the validation set; data of 18 patients can be assigned to the test set. In addition, all patient data can be exported as images with a resolution of 1024 pixels, where the training set can contain 2588 images, the validation set can contain 576 images, and the test set can contain 475 images. The network proposed in the present disclosure can be, but is not limited to: Python3.11.8, and all experiments can be performed under the PyTorch framework, and NVIDIA V100Tensor Core GPU can also be used.

[0068] In addition, in the present disclosure, the segmentation result of the blood vessel can be evaluated based on the average Dice coefficient and the average HD95, wherein the average Dice coefficient can be expressed as follows:

[0069]

[0070] Where X is the set of predicted values, Y is the set of true values, and n is the number of samples.

[0071] HD can be defined as follows:

[0072] Among them, 95% HD (HD95) is similar to the maximum HD, but it is based on calculating the 95th percentile of the distance between the boundary points between X and Y. The average HD95 can be defined as follows:

[0073] Here, n is the number of samples.

[0074] It should be noted that in the present disclosure, the proposed network can be trained using the Adam optimizer, the initial learning rate can be 0.001, and the aforementioned hybrid loss function can be used as a training target. In order to evaluate the performance of the model, Mean Dice) and Mean HD95 can be used as indicators. The experimental results can be shown in Table 1:

[0075] Table 1 Experimental results The test results show that the method provided by the present disclosure has achieved excellent performance in the segmentation results of DSA images (see Table 1 for details); and the experimental results show that the method provided by the present disclosure has excellent performance in both single-frame and sequence indicators. Specifically, the method provided by the present disclosure obtained the highest score in the single-frame average Dice and average HD95 evaluations. In addition, the method provided by the present disclosure also provides the best performance in the sequence mean Dice evaluation and obtains significantly favorable results in the sequence mean HD95 indicator.

[0076] Fig. 9 FIG. 4 is a schematic diagram showing a comparison between a single-frame prediction segmentation result of different blood vessel regions according to an exemplary embodiment of the present disclosure and a blood vessel fusion segmentation result obtained from a complete sequence. Fig. 9 ,The first row of the figure shows four frames uniformly extracted from the patient’s DSA image sequence, specifically frames 11, 21, 31, and 41. The last image shows that frame 24 was selected as a representative middle sequence frame to ,directly compare and visualize the results.

[0077] It can be observed from the single-frame segmentation results (Frame 1 and Frame 2) and the sequence fusion results (Sequence) that the method provided by the present disclosure achieves excellent performance in segmenting branch vessels, especially in the celiac trunk branches, superior mesenteric artery and renal artery. In the segmentation results of frames 1 and 2, there is an overlap between the superior mesenteric artery and the abdominal aorta, and only ScaleFormer and the method provided by the present disclosure successfully distinguish the two structures, thereby providing good segmentation results. For the segmentation of aneurysms and common iliac arteries, most methods show high accuracy. However, in the results of frame 4, Swin-Unet and DAEFormer show obvious under-segmentation problems. Regarding the segmentation of the right iliac artery, Swin-Unet shows obvious mis-segmentation in both frames 3 and 4. For the left iliac artery, only ScaleFormer and the method provided by the present disclosure achieve satisfactory segmentation results in frames 2, 3, 4 and the sequence results (Sequence). Further visualization of the results superimposed on the DSA image shows that the method provided by the present disclosure has the best segmentation performance, which is particularly evident in the case of complex branches and blurred blood vessel boundaries, that is, the method provided by the present disclosure effectively improves the integrity and accuracy of blood vessel segmentation.

[0078] The present invention provides a multi-frame fusion algorithm L-MSFUnet for abdominal aorta vessel segmentation based on DSA images. Compared with the existing methods, the present invention can effectively extract the complete vessel tree structure in the sequence image while segmenting the single-frame vessel, thereby effectively improving the integrity and accuracy of vessel segmentation.

[0079] Specifically, by introducing time frame encoding, the continuity of features across multiple frames can be enhanced; by adopting a feature extraction layer with a FreqPass module, the uniformity of blood vessels in intra-class images and inter-class differences can be effectively improved; by integrating the decoding structure of optical flow characteristics, the spatiotemporal continuity of blood vessels in image sequences can be ensured; by further fusing the segmentation results from different frames through the LSTM module, a comprehensive abdominal aorta vascular tree can be generated; the use of a hybrid channel attention module and a hybrid loss function significantly improves the segmentation accuracy. It can be seen that the vascular recognition model provided by the present disclosure can improve the accuracy of vascular extraction and provide more reliable decision support for clinical treatment.

[0080] Fig.10 is a flow chart showing a blood vessel recognition method according to an exemplary embodiment of the present disclosure, and the blood vessel recognition method can be implemented based on a blood vessel recognition model trained according to the training method of the present disclosure.

[0081] Reference Fig.10 In step 1001, a plurality of target blood vessel images to be identified may be acquired, wherein the plurality of target blood vessel images to be identified may be images obtained by injecting a contrast agent into a blood vessel of a biological body.

[0082] In step 1002, a plurality of target blood vessel images to be identified may be divided into a plurality of image groups to be identified, wherein the plurality of image groups to be identified may correspond one-to-one to a plurality of MSFUnet networks, and each image group to be identified may include at least two target blood vessel images to be identified.

[0083] It should be noted that the input image can also be divided into patches of size 5, and the step size of the division can be 4. Through this overlapping division operation, the edge effect of the image can be reduced. In addition, when encoding, in addition to encoding the position within the frame, the time information of the image can also be encoded according to the input order of the vascular image.

[0084] In step 1003, each image group to be identified may be input into the MSFUnet network corresponding to the image group to be identified, and a first blood vessel identification target result output by the MSFUnet network may be obtained.

[0085] In step 1004, a plurality of first blood vessel recognition target results corresponding to a plurality of MSFUnet networks may be input into the LSTM network to obtain a final blood vessel recognition target result.

[0086] Fig.11 1 is a block diagram showing a training apparatus 1100 for a blood vessel recognition model according to an exemplary embodiment of the present disclosure. The blood vessel recognition model may include an LSTM network and a plurality of MSFUnet networks.

[0087] Reference Fig.11 The training device 1100 for the blood vessel recognition model may include a blood vessel image acquisition module 1101, an image group division module 1102, a first blood vessel recognition result acquisition module 1103, a blood vessel recognition final result acquisition module 1104, a first loss calculation module 1105, a second loss calculation module 1106 and a training module 1107.

[0088] The blood vessel image acquisition module 1101 can acquire a plurality of blood vessel images, wherein the plurality of blood vessel images can be images obtained by injecting contrast agents into blood vessels of a living body.

[0089] The image group division module 1102 may divide the plurality of vascular images into a plurality of image groups, wherein the plurality of image groups may correspond one-to-one to a plurality of MSFUnet networks, and each image group may include at least two vascular images.

[0090] The first blood vessel recognition result acquisition module 1103 may input each image group into the MSFUnet network corresponding to the image group to obtain the first blood vessel recognition result output by the MSFUnet network.

[0091] According to an exemplary embodiment of the present disclosure, the MSFUnet network may include a double-attention module, a channel attention module (Channel Attention, CA), multiple encoder modules (SpatialFrequency Block, SFB), multiple residual convolution modules (resdecoder block, ResDe) and multiple optical flow modules (flow attention).

[0092] The first blood vessel identification result acquisition module 1103 may encode the image group through multiple encoder modules to obtain an encoding result. Then, the first blood vessel identification result acquisition module 1103 may input the encoding result into the dual attention module to obtain a first attention result. Next, the first blood vessel identification result acquisition module 1103 may perform optical flow processing on the first attention result through multiple optical flow modules to obtain an optical flow result.

[0093] Then, for the first residual convolution module among the multiple residual convolution modules, the first blood vessel recognition result acquisition module 1103 can obtain the residual convolution result of the residual convolution module based on the first attention result, the output of the encoder module at the same layer as the residual convolution module, and the output of the optical flow module at the same layer as the residual convolution module; for each residual convolution module from the second to the nth among the multiple residual convolution modules, the first blood vessel recognition result acquisition module 1103 can obtain the residual convolution result of the residual convolution module based on the output of the residual convolution module of the previous layer of the residual convolution module, the output of the encoder module at the same layer as the residual convolution module, and the output of the optical flow module at the same layer as the residual convolution module, where n can be an integer greater than or equal to 2.

[0094] Next, the first blood vessel identification result acquisition module 1103 can obtain the second attention result through the channel attention module based on the first attention result and the residual convolution result of each residual convolution module in the multiple residual convolution modules. Then, the first blood vessel identification result acquisition module 1103 can obtain the first blood vessel identification result of the MSFUnet network based on the residual convolution result output by the residual convolution module located in the last layer of the multiple residual convolution modules and the second attention result. Specifically, the first blood vessel identification result acquisition module 1103 can perform a Kronecker product operation on the residual convolution result output by the residual convolution module of the last layer and the second attention result, thereby obtaining the first blood vessel identification result of the MSFUnet network.

[0095] According to an exemplary embodiment of the present disclosure, each encoder module (SFB) may include a down-sampling module (FreqPass).

[0096] For the first encoder module, the first blood vessel recognition result acquisition module 1103 can obtain the down-sampled features of the average pooling branch (avg pool) and the down-sampled features of the maximum pooling branch (max pool) of the image group through the down-sampling module included in the encoder module, and can concatenate the down-sampled features of the average pooling branch and the down-sampled features of the maximum pooling branch to obtain the output result. Furthermore, the concatenation result of the down-sampled features can be operated using pointwise convolution to obtain the encoding result, that is, to obtain the final down-sampled feature Xdown (size is [b, 2c, h / 2, w / 2]).

[0097] For each encoder module except the first encoder module among the multiple encoder modules, the first blood vessel recognition result acquisition module 1103 can obtain the down-sampled features of the average pooling branch and the down-sampled features of the maximum pooling branch of the output result of the previous encoder module through the down-sampling module included in the encoder module, and can concatenate the down-sampled features of the average pooling branch and the down-sampled features of the maximum pooling branch to obtain the output result. In this way, the output result of the last encoder module among the multiple encoder modules can be used as the encoding result for inputting the double-attention module.

[0098] It should be noted that the aforementioned “down-sampling features of the average pooling branch” can be obtained in the following way: For the image group or the output of the previous encoder module, for example, the input feature X (size [b, c, h, w]) can be average pooled to obtain the average pooling result. Then, the average pooling result can be sequentially subjected to convolution operation, pixel rearrangement operation, and Softmax operation to obtain a feature with size [b, hw / 4, 3, 3]. Figure X s. Next, the Hamming window dot multiplication operation can be used to obtain the first convolution kernel Xw (size is [b, 9, h / 2, w / 2]). Then, the first convolution kernel Xw can be convolved with the average pooling result to obtain the down-sampled features of the average pooling branch.

[0099] The aforementioned "down-sampling features of the maximum pooling branch" can be obtained in the following way: For the image group or the output result of the previous encoder module, for example, the input feature X (size is [b,c,h,w]) can be subjected to the maximum pooling operation to obtain the maximum pooling result. Then, the maximum pooling result can be subjected to the convolution operation, the pixel rearrangement operation, and the Softmax operation in sequence to obtain the feature with the size of [b,hw / 4,3,3]. Figure X s. Next, the Hamming window dot multiplication operation can be used to obtain the second convolution kernel Xw (size is [b, 9, h / 2, w / 2]). Then, the second convolution kernel Xw can be convolved with the maximum pooling result to obtain the down-sampled features of the maximum pooling branch. In this way, the uniformity of blood vessels in intra-class images and the difference between classes can be effectively improved by using the feature extraction layer with the FreqPass module.

[0100] The blood vessel identification final result acquisition module 1104 may input a plurality of first blood vessel identification results corresponding one-to-one to a plurality of MSFUnet networks into the LSTM network to obtain a blood vessel identification final result.

[0101] The first loss calculation module 1105 can calculate the first loss based on the first blood vessel recognition result output by each MSFUnet network and the first image label bound to the image group corresponding to the MSFUnet network. The first image label can be an image label containing blood vessel morphology marked based on at least two blood vessel images included in the image group, that is, the first image label can be an image label containing blood vessel morphology marked by a doctor based on at least two blood vessel images included in the image group.

[0102] The second loss calculation module 1106 can calculate the second loss based on the final result of blood vessel recognition output by the LSTM network and the second image label bound to multiple blood vessel images, wherein the second image label can be an image label containing blood vessel morphology marked based on multiple blood vessel images.

[0103] Specifically, the second loss can be a sequence-level loss (pre_sequence, label_sequence) obtained by calculating the difference between the result of fusing the single-frame prediction image output by the MSFUnet network through the LSTM network module and the sequence label, and the loss can be calculated using Focal Loss. The aforementioned second image label can be an image label containing vascular morphology marked by a doctor based on multiple vascular images, that is, the doctor can mark the vascular morphology based on the multiple vascular images initially input into the vascular recognition model according to his own experience as the second image label.

[0104] The training module 1107 may train the blood vessel recognition model by adjusting the parameters of the MSFUnet network based on the first loss and adjusting the parameters of the LSTM network based on the second loss.

[0105] According to an exemplary embodiment of the present disclosure, the training device 1100 of the above-mentioned blood vessel recognition model may further include a third loss calculation module. For each MSFUnet network, the third loss calculation module may further calculate a third loss based on the optical flow result output by the optical flow module (flow) of the last layer of the multiple optical flow modules (flow) included in the MSFUnet network and the first image label. Next, the training module 1107 may adjust the parameters of the MSFUnet network based on both the aforementioned first loss and the third loss.

[0106] Fig.12 is a block diagram showing a blood vessel identification apparatus 1200 according to an exemplary embodiment of the present disclosure. The blood vessel identification apparatus 1200 may be implemented based on a blood vessel identification model trained according to the training method of the present disclosure.

[0107] Reference Fig.12 The blood vessel identification device 1200 may include a target blood vessel image acquisition module 1201, a to-be-identified image group division module 1202, a first blood vessel identification target result acquisition module 1203 and a blood vessel identification final target result acquisition module 1204.

[0108] The target blood vessel image acquisition module 1201 may acquire a plurality of target blood vessel images to be identified, wherein the plurality of target blood vessel images to be identified may be images obtained by injecting contrast agents into blood vessels of a living body.

[0109] The image group division module 1202 may divide multiple target blood vessel images to be identified into multiple image groups to be identified, wherein the multiple image groups to be identified may correspond one-to-one to multiple MSFUnet networks, and each image group to be identified may include at least two target blood vessel images to be identified.

[0110] It should be noted that the input image can also be divided into patches of size 5, and the step size of the division can be 4. Through this overlapping division operation, the edge effect of the image can be reduced. In addition, when encoding, in addition to encoding the position within the frame, the time information of the image can also be encoded according to the input order of the vascular image.

[0111] The first blood vessel recognition target result acquisition module 1203 may input each to-be-recognized image group into the MSFUnet network corresponding to the to-be-recognized image group to obtain the first blood vessel recognition target result output by the MSFUnet network.

[0112] The blood vessel identification final target result acquisition module 1204 may input a plurality of first blood vessel identification target results corresponding to a plurality of MSFUnet networks one by one into the LSTM network to obtain the blood vessel identification final target result.

[0113] Fig.13 is a block diagram illustrating an electronic device 1300 according to an exemplary embodiment of the present disclosure.

[0114] Reference Fig.13 The electronic device 1300 includes at least one memory 1301 and at least one processor 1302. The at least one memory 1301 stores instructions. When the instructions are executed by the at least one processor 1302, the training method of the blood vessel recognition model or the blood vessel recognition method according to the exemplary embodiment of the present disclosure is executed.

[0115] As an example, the electronic device 1300 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 1300 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 1300 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0116] In the electronic device 1300, the processor 1302 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0117] The processor 1302 may execute instructions or codes stored in the memory 1301, wherein the memory 1301 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0118] The memory 1301 may be integrated with the processor 1302, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 1301 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 1301 and the processor 1302 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 1302 can read files stored in the memory.

[0119] In addition, the electronic device 1300 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 1300 may be connected to each other via a bus and / or a network.

[0120] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the training method of the above-mentioned blood vessel recognition model or the blood vessel recognition method. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0121] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, wherein when the computer program is executed by a processor, the training method of the blood vessel recognition model or the blood vessel recognition method according to the present disclosure is implemented.

[0122] According to the training method of the vascular recognition model, the vascular recognition method, the device, the electronic device, the storage medium and the computer program product of the present invention, by using the multi-frame fusion algorithm L-MSFUnet, that is, by using multiple angiography images to perform vascular recognition, the amount of data considered is larger and the data content is richer, thereby ensuring the generation of a more comprehensive abdominal aorta vascular tree, that is, the accuracy of vascular recognition can be effectively improved, thereby effectively avoiding the occurrence of medical accidents.

[0123] According to an exemplary embodiment of the present disclosure, by introducing time frame coding, feature continuity across multiple frames can be enhanced.

[0124] According to an exemplary embodiment of the present disclosure, by setting a decoding structure that integrates optical flow characteristics, the spatiotemporal continuity of blood vessels in an image sequence can be ensured.

[0125] According to an exemplary embodiment of the present disclosure, the expressiveness of the feature channel may be enhanced through the CA module, thereby improving the representation capability of the network.

[0126] According to an exemplary embodiment of the present disclosure, the segmentation results from different frameworks can be further fused through an LSTM network module, thereby generating a comprehensive abdominal aorta vascular tree.

[0127] According to an exemplary embodiment of the present disclosure, the uniformity of blood vessels in intra-class images and the difference between classes can be effectively improved by using a feature extraction layer with a FreqPass module.

[0128] According to an exemplary embodiment of the present disclosure, the segmentation accuracy of blood vessels can be significantly improved by using a hybrid loss function.

[0129] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a blood vessel recognition model, characterized in that: The blood vessel recognition model includes an LSTM network and multiple MSFUnet networks, and the training method includes: Acquiring a plurality of vascular images, wherein the plurality of vascular images are images obtained by injecting a contrast agent into a blood vessel of a living body; Dividing the plurality of vascular images into a plurality of image groups, wherein the plurality of image groups correspond one-to-one to the plurality of MSFUnet networks, and each image group contains at least two of the vascular images; Input each image group into the MSFUnet network corresponding to the image group to obtain a first blood vessel recognition result output by the MSFUnet network; Inputting a plurality of first blood vessel recognition results corresponding to the plurality of MSFUnet networks one by one into the LSTM network to obtain a final blood vessel recognition result; Calculate a first loss based on a first blood vessel recognition result output by each MSFUnet network and a first image label bound to an image group corresponding to the MSFUnet network, wherein the first image label is an image label containing blood vessel morphology annotated based on at least two blood vessel images included in the image group; Calculating a second loss based on the final result of blood vessel recognition output by the LSTM network and a second image label bound to the plurality of blood vessel images, wherein the second image label is an image label containing blood vessel morphology annotated based on the plurality of blood vessel images; The blood vessel recognition model is trained by adjusting the parameters of the MSFUnet network based on the first loss and adjusting the parameters of the LSTM network based on the second loss.

2. The training method according to claim 1, characterized in that: The MSFUnet network includes a dual attention module, a channel attention module, multiple encoder modules, multiple residual convolution modules and multiple optical flow modules; The step of inputting each image group into the MSFUnet network corresponding to the image group to obtain a first blood vessel recognition result output by the MSFUnet network includes: Encoding the image group through the multiple encoder modules to obtain an encoding result; Inputting the encoding result into the dual attention module to obtain a first attention result; Performing optical flow processing on the first attention result through the multiple optical flow modules to obtain an optical flow result; For a first residual convolution module among the multiple residual convolution modules, based on the first attention result, an output of an encoder module at the same layer as the residual convolution module, and an output of an optical flow module at the same layer as the residual convolution module, obtaining a residual convolution result of the residual convolution module; For each residual convolution module from the second to the nth of the multiple residual convolution modules, based on the output of the residual convolution module of the previous layer of the residual convolution module, the output of the encoder module of the same layer as the residual convolution module, and the output of the optical flow module of the same layer as the residual convolution module, obtain the residual convolution result of the residual convolution module, where n is an integer greater than or equal to 2; Obtaining, by the channel attention module, a second attention result based on the first attention result and the residual convolution result of each residual convolution module in the plurality of residual convolution modules; Based on the residual convolution result output by the residual convolution module located at the last layer among the multiple residual convolution modules and the second attention result, a first blood vessel recognition result of the MSFUnet network is obtained.

3. The training method according to claim 2, characterized in that: The training method further comprises: For each MSFUnet network, based on the optical flow result output by the optical flow module located at the last layer among the multiple optical flow modules included in the MSFUnet network and the first image label, calculate the third loss; The adjusting the parameters of the MSFUnet network based on the first loss comprises: By adjusting the parameters of the MSFUnet network based on both the first loss and the third loss.

4. The training method according to claim 2, characterized in that: Each encoder module contains a downsampling module; The step of encoding the image group by using the multiple encoder modules to obtain an encoding result includes: For the first encoder module, obtain the down-sampling features of the average pooling branch and the down-sampling features of the maximum pooling branch of the image group through the down-sampling module included in the encoder module, and splice the down-sampling features of the average pooling branch and the down-sampling features of the maximum pooling branch to obtain an output result; For each encoder module except the first encoder module among the multiple encoder modules, obtain the down-sampling features of the average pooling branch and the down-sampling features of the maximum pooling branch of the output result of the previous encoder module through the down-sampling module included in the encoder module, and splice the down-sampling features of the average pooling branch and the down-sampling features of the maximum pooling branch to obtain an output result; Using the output result of the last encoder module among the multiple encoder modules as the encoding result; The down-sampling features of the average pooling branch are obtained in the following way: Perform an average pooling operation on the image group or the output result of the previous encoder module to obtain an average pooling result; perform a convolution operation, a pixel rearrangement operation, a Softmax operation, and a Hamming window dot multiplication operation on the average pooling result in sequence to obtain a first convolution kernel; convolve the first convolution kernel with the average pooling result to obtain a down-sampling feature of the average pooling branch; The down-sampling features of the maximum pooling branch are obtained in the following way: Perform a maximum pooling operation on the image group or the output result of the previous encoder module to obtain a maximum pooling result; perform a convolution operation, a pixel rearrangement operation, a Softmax operation, and a Hamming window dot multiplication operation on the maximum pooling result in sequence to obtain a second convolution kernel; convolve the second convolution kernel with the maximum pooling result to obtain a downsampling feature of the maximum pooling branch.

5. A blood vessel identification method, characterized in that: The blood vessel recognition method is implemented based on a blood vessel recognition model trained by a training method according to any one of claims 1 to 4, and the blood vessel recognition method includes: Acquire a plurality of target blood vessel images to be identified, wherein the plurality of target blood vessel images to be identified are images obtained by injecting contrast agents into blood vessels of a living body; Dividing the plurality of target blood vessel images to be identified into a plurality of image groups to be identified, wherein the plurality of image groups to be identified correspond one-to-one to the plurality of MSFUnet networks, and each image group to be identified includes at least two target blood vessel images to be identified; Input each image group to be identified into the MSFUnet network corresponding to the image group to be identified, and obtain the first blood vessel identification target result output by the MSFUnet network; A plurality of first blood vessel recognition target results corresponding one-to-one to the plurality of MSFUnet networks are input into the LSTM network to obtain a final blood vessel recognition target result.

6. A training device for a blood vessel recognition model, characterized in that: The blood vessel recognition model includes an LSTM network and multiple MSFUnet networks, and the training device includes: A blood vessel image acquisition module is configured to acquire a plurality of blood vessel images, wherein the plurality of blood vessel images are images obtained by injecting a contrast agent into a blood vessel of a living body; An image group division module is configured to divide the plurality of blood vessel images into a plurality of image groups, wherein the plurality of image groups correspond one-to-one to the plurality of MSFUnet networks, and each image group contains at least two of the blood vessel images; A first blood vessel recognition result acquisition module is configured to input each image group into the MSFUnet network corresponding to the image group, and obtain a first blood vessel recognition result output by the MSFUnet network; a blood vessel identification final result acquisition module, configured to input a plurality of first blood vessel identification results corresponding to the plurality of MSFUnet networks one by one into the LSTM network to obtain a blood vessel identification final result; A first loss calculation module is configured to calculate a first loss based on a first blood vessel recognition result output by each MSFUnet network and a first image label bound to an image group corresponding to the MSFUnet network, wherein the first image label is an image label containing blood vessel morphology annotated based on at least two blood vessel images included in the image group; A second loss calculation module is configured to calculate a second loss based on the final result of blood vessel recognition output by the LSTM network and a second image label bound to the plurality of blood vessel images, wherein the second image label is an image label containing blood vessel morphology annotated based on the plurality of blood vessel images; A training module is configured to train the blood vessel recognition model by adjusting the parameters of the MSFUnet network based on the first loss and adjusting the parameters of the LSTM network based on the second loss.

7. A blood vessel identification device, characterized in that: The blood vessel recognition device is implemented based on a blood vessel recognition model trained by the training method according to any one of claims 1 to 4, and the blood vessel recognition device includes: a target blood vessel image acquisition module, configured to acquire a plurality of target blood vessel images to be identified, wherein the plurality of target blood vessel images to be identified are images obtained by injecting a contrast agent into a blood vessel of a living body; The to-be-recognized image group division module is configured to divide the plurality of to-be-recognized target blood vessel images into a plurality of to-be-recognized image groups, wherein the plurality of to-be-recognized image groups correspond one-to-one to the plurality of MSFUnet networks, and each to-be-recognized image group includes at least two of the to-be-recognized target blood vessel images; A first blood vessel recognition target result acquisition module is configured to input each to-be-recognized image group into the MSFUnet network corresponding to the to-be-recognized image group, and obtain a first blood vessel recognition target result output by the MSFUnet network; The blood vessel identification final target result acquisition module is configured to input a plurality of first blood vessel identification target results corresponding to the plurality of MSFUnet networks one by one into the LSTM network to obtain a blood vessel identification final target result.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the blood vessel recognition model training method according to any one of claims 1 to 4, or to implement the blood vessel recognition method according to claim 5.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the blood vessel recognition model training method as described in any one of claims 1 to 4, or execute the blood vessel recognition method as described in claim 5.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method for training a blood vessel recognition model according to any one of claims 1 to 4, or implements the method for blood vessel recognition according to claim 5.

Images