Cross-view blood flow imaging prediction method, device, equipment and medium

Abstract

The present application relates to a cross-view blood flow imaging prediction method, device, equipment and medium, wherein the method includes: receiving a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocessing the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image; inputting the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of the target view; superimposing and fusing the basic two-dimensional ultrasonic structure prior image and the blood flow imaging prediction result of the target view to obtain a Doppler color ultrasonic image of the target view. The present application realizes cross-view prediction of cardiac blood flow imaging, and provides a new way to obtain cardiac blood flow imaging.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cross-view blood flow imaging prediction method, device, equipment and medium. Background Art

[0002] In clinical practice, color Doppler echocardiography has become the first-line imaging method for diagnosing heart diseases because it can display blood flow information within the heart. However, visual inspection and manual measurement of clinical parameters from multi-view ultrasound images is a time-consuming and laborious process. Although artificial intelligence can be used to automate this process, a frequent challenge is the lack of views. Most of the existing machine learning-based Doppler ultrasound models for diagnosing heart diseases use a single view as input, which may lead to missing important information.

[0003] Using deep learning methods to predict blood flow images of any target view from the source view can alleviate the problems of incomplete and inaccurate diagnosis and evaluation caused by missing views; especially for moderate to severe cases, comprehensive evaluation and diagnosis are important prerequisites for choosing the timing of surgical intervention. The only existing method for synthesizing cardiac color Doppler is the vector field method, in which the synthesized blood vector field is obtained from a patient-specific computational fluid dynamics model, and the real texture and clutter artifacts are simulated from real clinical ultrasound images. Therefore, the current cross-view prediction of cardiac blood flow imaging technology is still blank. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a cross-view blood flow imaging prediction method, device, equipment and medium to achieve cross-view prediction of cardiac blood flow imaging.

[0005] In order to solve the above technical problems, the present application embodiment provides a cross-view blood flow imaging prediction method, including:

[0006] Receiving a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocessing the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image;

[0007] Inputting the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of a target view;

[0008] The basic two-dimensional ultrasound structure prior image and the blood flow imaging prediction result of the target view are superimposed and fused to obtain a Doppler color ultrasound image of the target view.

[0009] In order to solve the above technical problems, the present application provides a cross-view blood flow imaging prediction device, including:

[0010] An image receiving unit is used to receive a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocess the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image;

[0011] A blood flow prediction unit, used for inputting the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of a target view;

[0012] The superposition and fusion unit is used to superimpose and fuse the basic two-dimensional ultrasonic structure prior image and the blood flow imaging prediction result of the target view to obtain a Doppler color ultrasonic image of the target view.

[0013] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that the one or more processors can implement any one of the above-mentioned cross-view blood flow imaging prediction methods.

[0014] In order to solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned cross-view blood flow imaging prediction methods is implemented.

[0015] The embodiments of the present invention provide a cross-view blood flow imaging prediction method, device, equipment and medium. The method includes: receiving a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocessing the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image; inputting the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of the target view; superimposing and fusing the basic two-dimensional ultrasonic structure prior image and the blood flow imaging prediction result of the target view to obtain a Doppler color ultrasonic image of the target view. The embodiments of the present invention realize cross-view prediction of cardiac blood flow imaging, and provide a new way to obtain cardiac blood flow imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of the implementation of the cross-view blood flow imaging prediction method provided in the embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of the prediction process of the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a cross-view learning network framework of a cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0020] Figure 4 It is a schematic diagram of the Doppler color blood flow imaging result of the cross-view blood flow imaging prediction method provided by the embodiment of the present application in any target view;

[0021] Figure 5 It is a flowchart for implementing a sub-process in the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0022] Figure 6 It is a flowchart for implementing a sub-process in the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0023] Figure 7 It is a flowchart for implementing a sub-process in the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0024] Figure 8 It is a flowchart for implementing a sub-process in the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0025] Fig. 9 It is a flowchart for implementing a sub-process in the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0026] Fig.10 It is a flowchart for implementing a sub-process in the cross-view blood flow imaging prediction method provided in an embodiment of the present application;

[0027] Fig.11 is a schematic diagram of a cross-view blood flow imaging prediction device provided in an embodiment of the present application;

[0028] Fig.12 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0030] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0032] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the cross-view blood flow imaging prediction method provided in the embodiment of the present application is generally executed by a server, and accordingly, the cross-view blood flow imaging prediction device is generally configured in the server.

[0034] See also Figures 1 to 4 , Figure 1 A specific implementation of a cross-view blood flow imaging prediction method is shown; Figure 2 It is a schematic diagram of the prediction process of the cross-view blood flow imaging prediction method provided in an embodiment of the present application; Figure 3 It is a schematic diagram of a cross-view learning network framework of a cross-view blood flow imaging prediction method provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the Doppler color blood flow imaging results of the cross-view blood flow imaging prediction method provided in an embodiment of the present application in any target view.

[0035] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 1 The process sequence shown is limited to the following steps:

[0036] S1: receiving a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocessing the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image.

[0037] Specifically, the embodiment of the present application predicts the blood flow image of any target view from the source view by a deep learning method, so it is necessary to receive the Doppler color image of the source view and the two-dimensional ultrasound structure prior image of the target view. In the embodiment of the present application, the preprocessing of the Doppler color image and the two-dimensional ultrasound structure prior image includes image size unification and image pixel normalization.

[0038] See also Figure 5 , Figure 5 A specific implementation of step S1 is shown, which is described in detail as follows:

[0039] S11: Receive the Doppler color image of the source view and the two-dimensional ultrasound structure prior image of the target view.

[0040] S12: cropping the Doppler color image and the two-dimensional ultrasonic structure prior image to the same size to obtain a cropped Doppler color image and a cropped two-dimensional ultrasonic structure prior image.

[0041] S13: performing pixel normalization processing on the cropped Doppler color image and the cropped two-dimensional ultrasonic structure prior image to obtain the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image.

[0042] Specifically, the embodiment of the present application needs to crop the Doppler color image and the two-dimensional ultrasound structure prior image to the same size, and the size of the size can be set according to the actual situation and is not limited here. In a specific embodiment, the Doppler color image and the two-dimensional ultrasound structure prior image are cropped to 224×224. Then, the cropped Doppler color image and the cropped two-dimensional ultrasound structure prior image are pixel normalized to the [0, 1] interval.

[0043] S2: Inputting the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of the target view.

[0044] Specifically, before performing cardiac blood flow imaging prediction, it is necessary to first construct a cross-view learning network, and train and optimize the cross-view learning network to obtain a target view learning network. Therefore, when blood flow imaging prediction is required, the basic Doppler color image and the basic two-dimensional ultrasound structure prior image preprocessed in step S1 are input into the pre-trained cross-view learning network for prediction processing, thereby obtaining the blood flow imaging prediction result of the target view. Among them, the cross-view learning network is a Transformer-based neural network, which includes three parts: a shared encoder module, a multi-scale cross-attention mechanism module, and a decoder module.

[0045] See also Figure 6 , Figure 6 A specific implementation of step S2 is shown, which is described in detail as follows:

[0046] S21: Inputting the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into the pre-trained cross-view learning network.

[0047] S22: extracting multi-scale features of the basic Doppler color image and the basic two-dimensional ultrasound structure prior image through the encoder module to obtain source view features and target view structure prior features.

[0048] See also Figure 7 , Figure 7 A specific implementation of step S22 is shown, which is described in detail as follows:

[0049] S221: In the encoder module, the basic Doppler color image and the basic two-dimensional ultrasound structure prior image are divided into non-overlapping image blocks and embedded in a linear space, and the self-attention of the image blocks is calculated.

[0050] S222: Merge the connected image blocks and calculate the self-attention of the merged image blocks.

[0051] S223: Generate the source view feature and the target view structure prior feature based on the self-attention of the image block and the self-attention of the merged image block.

[0052] Specifically, the shared encoder module consists of four stages. The first stage divides the image into non-overlapping blocks and embeds them into a linear space, and calculates self-attention through the Swin Transformer Block. The next three stages first merge adjacent image blocks and calculate self-attention. Different levels of representation are generated through the four stages, so that source view features and target view structural prior features can be generated. The basic Doppler color image and the basic two-dimensional ultrasound structural prior image are respectively extracted with multi-scale feature maps through the shared encoder.

[0053] S23: In the multi-scale cross-attention mechanism module, multi-scale cross-attention calculation is performed on the source view features and the target view structure prior features to obtain a multi-scale fusion feature map.

[0054] See also Figure 8 , Figure 8 A specific implementation of step S23 is shown, which is described in detail as follows:

[0055] S231: In the multi-scale cross-attention mechanism module, up-sample the keys and values ​​of the high-level semantic features in the source view features to obtain the keys and values ​​of the multi-scale low-level semantic features.

[0056] S232: taking the target view structure prior features as a query, and performing multi-scale cross-attention calculations in combination with the keys and values ​​of the multi-scale low-level semantic features to obtain the multi-scale fusion feature map.

[0057] Specifically, the multi-scale cross-attention mechanism module is composed of a cross-attention mechanism layer and an upsampling layer. In the embodiment of the present application, the keys and values ​​of the high-level semantic features in the source view features are upsampled to obtain the keys and values ​​of the multi-scale low-level semantic features; the target view structure prior features are used as queries, and the multi-scale cross-attention calculation is performed in combination with the keys and values ​​of the multi-scale low-level semantic features to obtain a multi-scale fusion feature map.

[0058] In an embodiment of the present application, for a high-level semantic feature map, the structural prior features of the target view are used as the query in the attention mechanism, and the source view features are used as the key and value in the attention mechanism; in order to efficiently and effectively calculate the multi-scale cross-attention, for a larger-scale feature map, the keys and values ​​of the high-level semantic features are upsampled to obtain large-scale keys and values, and multi-scale cross-attention calculations are performed to finally obtain a multi-scale fused feature map.

[0059] S24: upsampling and skip connection processing are performed on the multi-scale fusion feature map through the decoder module to decode the multi-scale fusion feature map to obtain a blood flow imaging prediction result of the target view.

[0060] Specifically, the decoder module consists of upsampling, skip connection and self-attention layers. The multi-scale fusion feature map is further integrated with feature maps of different scales through upsampling layers and skip connections, and then passes through 4 layers of Swin Transformer Block with powerful long-range dependency modeling capabilities to enhance global perception and better reconstruct blood flow information, thereby obtaining the blood flow imaging prediction results of the target view.

[0061] See also Fig. 9 , Fig. 9 A specific implementation method before step S2 is shown, which is described in detail as follows:

[0062] S2A: Acquires an initial Doppler color image of the source view and an initial 2D ultrasound image of the target view.

[0063] Specifically, an initial Doppler color image of the source view and an initial two-dimensional ultrasound image of the target view are collected, and these images are used as data sets for model training.

[0064] In a specific embodiment, the dataset used is from a tertiary hospital, with a total of 337 patients with mitral regurgitation. Each patient has a maximum of 5 different views of Doppler color echocardiograms. The views included in the dataset are: parasternal long axis (PLAX), apical four-chamber (A4C), apical three-chamber (A3C), apical two-chamber (A2C) and apical five-chamber (A5C). The regurgitation grade distribution is (mild 168 cases, moderate 79 cases, severe 90 cases). All patients were randomly divided into training sets (n = 228), validation sets (n = 38) and test sets (n = 71). All cases were annotated by senior doctors with view categories, maximum frames of mitral regurgitation, and regurgitation grade labels. It is worth noting that most patients have the same view containing the results of multiple scans. In the training stage, a random strategy is used to select one from multiple videos of each identical view. This method allows more additional samples to be generated for fair comparison, validation and testing, and a fixed strategy is used to select the first one.

[0065] S2B: Preprocessing the initial Doppler color image and the initial two-dimensional ultrasound image to obtain a preprocessed initial Doppler color image and a preprocessed initial two-dimensional ultrasound image.

[0066] Specifically, the initial Doppler color image and the initial two-dimensional ultrasound image are preprocessed so that the image sizes of the initial Doppler color image and the initial two-dimensional ultrasound image are uniformly scaled to 224*224, and the image pixels are normalized to the interval [0, 1].

[0067] S2C: extracting color labels and a structural prior image of a target view from the preprocessed initial two-dimensional ultrasound image.

[0068] See also Fig.10 , Fig.10 A specific implementation of step S2C is shown, which is described in detail as follows:

[0069] S2C1: extracting the blood flow frame contour of the preprocessed initial two-dimensional ultrasound image based on a contour detection algorithm.

[0070] S2C2: a square ROI region is cropped from the blood flow frame outline, and the ROI region is uniformly cropped to a preset size using a bilinear interpolation method to obtain a cropped image.

[0071] S2C3: Convert the cropped image from the RGB color space to the HSV color space to obtain a color-converted image.

[0072] S2C4: Segment the color-converted image using a threshold segmentation method to obtain the color mask.

[0073] S2C5: Obtain the color label by performing an AND operation on the color mask and the basic Doppler color image.

[0074] S2C6: The structural prior image of the target view is obtained by performing a negation operation on the color mask and performing an AND operation on the basic Doppler color image, and converting the result into a grayscale image.

[0075] Specifically, the above-mentioned preprocessed initial Doppler color image and initial two-dimensional ultrasound image have corresponding annotations. The embodiment of the present application needs to sample the largest annotated frame from these images. In order to prevent the image from being deformed, the blood flow box contour is first extracted according to the contour detection algorithm, and then the ROI (Region of Interest) area is cropped out with a square, and then the cropped image is uniformly scaled to 224*224 using bilinear interpolation. Then, by converting the target view from RGB to HSV color space, the color-converted image is segmented using the threshold segmentation method to obtain a color mask; then, the color mask and the basic Doppler color image are ANDed to obtain a color label; then, the color mask is negated and ANDed with the basic Doppler color image, and converted into a grayscale image to obtain a structural prior image of the target view.

[0076] S2D: constructing a cross-view learning network, and training the cross-view learning network based on the preprocessed initial Doppler color image, the color label and the structural prior image of the target view to obtain the pre-trained cross-view learning network.

[0077] Specifically, the details of the cross-view learning network are shown in Table 1:

[0078] Table 1 Cross-view learning network details

[0079]

[0080]

[0081] Specifically, in Table 1, the upsampling method is bilinear interpolation on the two-dimensional feature map, which does not change the depth of the feature map. When calculating the multi-scale cross attention, the high-level semantic feature maps extracted from the structure prior maps of the source view and the target view (the output of the last layer of the encoder) are F s ∈R wh×768 , F t ∈R wh×768 ; The fused feature map F is calculated using the attention mechanism shown in the following formula 1 f ∈R wh×768 In order to improve computational efficiency, the keys and values ​​of the larger-scale feature maps are not obtained through matrix projection, which requires large computational effort, but through upsampling of the keys and values ​​corresponding to the smaller-scale feature maps. The fused multi-scale feature maps obtained through the multi-scale cross-attention module and the decoder use jump connections to merge feature maps of the same size, thereby improving the performance of the model prediction. Formula (1) is as follows:

[0082]

[0083] Among them, W q , W k and W s They represent the linear projection matrix, d k Yes F t Dimension after linear projection.

[0084] Furthermore, the Python programming language and the Pytorch deep learning framework are used to build the cross-view learning network in Table 1, in which the encoder is initialized with parameters pre-trained in ImageNet1K, which is conducive to the training and convergence of the model. Other modules are initialized with Pytoch custom parameters. In the embodiment of the present application, the cross-view learning network adopts a self-supervised training method, the loss function is the MSE at the pixel level, the Batch size is set to 64, and the training cycle is 5000; the experiment uses the AdamW optimizer, and sets the attenuation coefficient to 0.05; the initial learning rate is 5e-4; and the cosine decay is used to 0.1 times the initial value.

[0085] S3: superimposing and fusing the basic two-dimensional ultrasound structure prior image and the blood flow imaging prediction result of the target view to obtain a Doppler color ultrasound image of the target view.

[0086] Specifically, the above steps have obtained the blood flow imaging prediction result of the target view. In the embodiment of the present application, the blood flow imaging prediction result of the target view is superimposed with the basic two-dimensional ultrasound structure prior image to obtain the Doppler color ultrasound image of the target view, that is, the final blood flow imaging of the target view is obtained.

[0087] See also Figure 4 As shown, Figure 4 Schematic diagram of the Doppler color blood flow imaging results of the cross-view blood flow imaging prediction method provided by the embodiment of the present application in any target view. Figure 4 There are 5 views in the image, so there are 20 ways to predict the target view from the source view, including: top left: source view used to predict blood flow imaging, each row corresponds to a source view; top right: target view is the actual situation, each column corresponds to a target view; bottom: blood flow imaging results of the target view predicted from the source view, the source view is each row label, and the target view is each column label. Although there are large differences between the source view and the target view, the algorithm can accurately understand the spatial relationship between different views, thereby accurately predicting the blood flow imaging of the target view.

[0088] In an embodiment of the present application, a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view are received, and the Doppler color image and the two-dimensional ultrasonic structure prior image are preprocessed respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image; the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image are input into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of the target view; the basic two-dimensional ultrasonic structure prior image and the blood flow imaging prediction result of the target view are superimposed and fused to obtain a Doppler color ultrasonic image of the target view. The embodiment of the present invention realizes cross-view prediction of cardiac blood flow imaging, and provides a new way to obtain cardiac blood flow imaging.

[0089] Furthermore, the embodiment of the present application predicts the blood flow imaging of the target view based on the structural priors of the source view and the target view, so that a single model can predict the blood flow imaging effects of any number of views, and can flexibly deal with the problem of random view missing in reality. In the interaction of different view features, the embodiment of the present application designs a more efficient calculation method based on the cross-attention mechanism. For large-scale feature maps, the calculation of the cross-attention mechanism faces the problem of large amount of computation. The attention map of a higher semantic layer is directly upsampled and applied to the low-level feature map to optimize this process, thereby improving the reasoning speed of the model and the prediction performance. The blood flow imaging predicted by the embodiment of the present application has good clinical fidelity, and the accuracy of mitral regurgitation grading has been improved in verification experiments, demonstrating its potential value in clinical applications.

[0090] Please refer to Fig.11 , as a response to the above Figure 1 The present application provides an embodiment of a cross-view blood flow imaging prediction device, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0091] like Fig.11 As shown, the cross-view blood flow imaging prediction device of this embodiment includes: an image receiving unit 41, a blood flow prediction unit 42 and a superposition fusion unit 43, wherein:

[0092] The image receiving unit 41 is used to receive the Doppler color image of the source view and the two-dimensional ultrasonic structure prior image of the target view, and preprocess the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image;

[0093] A blood flow prediction unit 42 is used to input the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of a target view;

[0094] The superposition and fusion unit 43 is used to superimpose and fuse the basic two-dimensional ultrasonic structure priori image and the blood flow imaging prediction result of the target view to obtain a Doppler color ultrasonic image of the target view.

[0095] Furthermore, the pre-trained cross-view learning network includes an encoder module, a multi-scale cross-attention mechanism module and a decoder module, and the blood flow prediction unit 42 includes:

[0096] An image input unit, used for inputting the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image into the pre-trained cross-view learning network;

[0097] A feature extraction unit, configured to extract multi-scale features of the basic Doppler color image and the basic two-dimensional ultrasound structure prior image through the encoder module to obtain source view features and target view structure prior features;

[0098] A first attention calculation unit, configured to perform multi-scale cross-attention calculation on the source view feature and the target view structure prior feature in the multi-scale cross-attention mechanism module to obtain a multi-scale fusion feature map;

[0099] A feature decoding unit is used to perform upsampling and jump connection processing on the multi-scale fusion feature map through the decoder module to decode the multi-scale fusion feature map to obtain a blood flow imaging prediction result of the target view.

[0100] Furthermore, the feature extraction unit comprises:

[0101] A second attention calculation unit is used for dividing the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into non-overlapping image blocks and embedding them into a linear space in the encoder module, and calculating the self-attention of the image blocks;

[0102] An image block merging unit, used to merge the connected image blocks and calculate the self-attention of the merged image blocks;

[0103] A feature generation unit is used to generate the source view feature and the target view structure prior feature based on the self-attention of the image block and the self-attention of the merged image block.

[0104] Furthermore, the first attention calculation unit includes:

[0105] An upsampling unit, configured to perform upsampling processing on the keys and values ​​of the high-level semantic features in the source view features in the multi-scale cross-attention mechanism module to obtain the keys and values ​​of the multi-scale low-level semantic features;

[0106] The third attention calculation unit is used to use the target view structure prior features as a query, and perform multi-scale cross-attention calculations in combination with the keys and values ​​of the multi-scale low-level semantic features to obtain the multi-scale fusion feature map.

[0107] Furthermore, the blood flow prediction unit 42 further includes:

[0108] An image acquisition unit, used for acquiring an initial Doppler color image of a source view and an initial two-dimensional ultrasound image of a target view;

[0109] A preprocessing unit, used for preprocessing the initial Doppler color image and the initial two-dimensional ultrasonic image to obtain a preprocessed initial Doppler color image and a preprocessed initial two-dimensional ultrasonic image;

[0110] A color label unit, used for extracting a color label and a structural prior image of a target view from the preprocessed initial two-dimensional ultrasound image;

[0111] A network construction unit is used to construct a cross-view learning network, and train the cross-view learning network based on the preprocessed initial Doppler color image, the color label and the structural prior image of the target view to obtain the pre-trained cross-view learning network.

[0112] Furthermore, the color label unit comprises:

[0113] A contour extraction unit, used for extracting the blood flow frame contour of the preprocessed initial two-dimensional ultrasonic image based on a contour detection algorithm;

[0114] A first image cropping unit, used for cropping a square ROI area from the blood flow frame outline, and using a bilinear interpolation method to uniformly crop the ROI area to a preset size to obtain a cropped image;

[0115] A color space conversion unit, used to convert the cropped image from the RGB color space to the HSV color space to obtain a color-converted image;

[0116] A color mask generating unit, configured to segment the color-converted image using a threshold segmentation method to obtain the color mask;

[0117] A color label generating unit, configured to obtain the color label by performing an AND operation on the color mask and the basic Doppler color image;

[0118] The structural prior image generating unit is used for obtaining the structural prior image of the target view by performing a negation operation on the color mask and an AND operation on the basic Doppler color image, and converting the result into a grayscale image.

[0119] Furthermore, the image receiving unit 41 includes:

[0120] An image receiving unit, configured to receive the Doppler color image of the source view and the two-dimensional ultrasonic structure prior image of the target view;

[0121] A second image cropping unit is used to crop the Doppler color image and the two-dimensional ultrasonic structure prior image to the same size to obtain a cropped Doppler color image and a cropped two-dimensional ultrasonic structure prior image;

[0122] The pixel normalization unit is used to perform pixel normalization processing on the cropped Doppler color image and the cropped two-dimensional ultrasonic structure prior image to obtain the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image.

[0123] In an embodiment of the present application, a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view are received, and the Doppler color image and the two-dimensional ultrasonic structure prior image are preprocessed respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image; the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image are input into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of the target view; the basic two-dimensional ultrasonic structure prior image and the blood flow imaging prediction result of the target view are superimposed and fused to obtain a Doppler color ultrasonic image of the target view. The embodiment of the present invention realizes cross-view prediction of cardiac blood flow imaging, and provides a new way to obtain cardiac blood flow imaging.

[0124] To solve the above technical problems, the present application also provides a computer device. Fig.12 , Fig.12 This is a basic structural block diagram of the computer device in this embodiment.

[0125] The computer device 5 includes a memory 51, a processor 52, and a network interface 53 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 5 having three components: a memory 51, a processor 52, and a network interface 53, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0126] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0127] The memory 51 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 51 can be an internal storage unit of the computer device 5, such as a hard disk or memory of the computer device 5. In other embodiments, the memory 51 can also be an external storage device of the computer device 5, such as a plug-in hard disk equipped on the computer device 5, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 51 can also include both the internal storage unit of the computer device 5 and its external storage device. In this embodiment, the memory 51 is generally used to store the operating system and various application software installed on the computer device 5, such as the program code of the cross-view blood flow imaging prediction method, etc. In addition, the memory 51 can also be used to temporarily store various types of data that have been output or are to be output.

[0128] The processor 52 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 52 is generally used to control the overall operation of the computer device 5. In this embodiment, the processor 52 is used to run the program code stored in the memory 51 or process data, such as running the program code of the above-mentioned cross-view blood flow imaging prediction method to implement various embodiments of the cross-view blood flow imaging prediction method.

[0129] The network interface 53 may include a wireless network interface or a wired network interface, and the network interface 53 is generally used to establish a communication connection between the computer device 5 and other electronic devices.

[0130] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores a computer program. The computer program can be executed by at least one processor to enable the at least one processor to perform the steps of a cross-view blood flow imaging prediction method as described above.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0132] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A cross-view blood flow imaging prediction method, characterized in that: include: Receiving a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocessing the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image; Inputting the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of a target view; The basic two-dimensional ultrasound structure prior image and the blood flow imaging prediction result of the target view are superimposed and fused to obtain a Doppler color ultrasound image of the target view; The pre-trained cross-view learning network includes an encoder module, a multi-scale cross-attention mechanism module and a decoder module. The basic Doppler color image and the basic two-dimensional ultrasound structure prior image are input into the pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of the target view, including: Inputting the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into the pre-trained cross-view learning network; In the encoder module, the basic Doppler color image and the basic two-dimensional ultrasound structure prior image are divided into non-overlapping image blocks and embedded in a linear space, and the self-attention of the image blocks is calculated; Merging the connected image blocks and calculating the self-attention of the merged image blocks; Generate source view features and target view structure prior features based on the self-attention of the image block and the self-attention of the merged image block; In the multi-scale cross-attention mechanism module, upsampling is performed on the keys and values ​​of the high-level semantic features in the source view features to obtain the keys and values ​​of the multi-scale low-level semantic features; The target view structure prior feature is used as a query, and a multi-scale cross-attention calculation is performed in combination with the key and value of the multi-scale low-level semantic feature to obtain a multi-scale fused feature map; The multi-scale fusion feature map is upsampled and skip-connected through the decoder module to decode the multi-scale fusion feature map to obtain a blood flow imaging prediction result of the target view.

2. The cross-view blood flow imaging prediction method according to claim 1, characterized in that: Before inputting the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of a target view, the method further includes: acquiring an initial Doppler color image of a source view and an initial two-dimensional ultrasound image of a target view; Preprocessing the initial Doppler color image and the initial two-dimensional ultrasonic image to obtain a preprocessed initial Doppler color image and a preprocessed initial two-dimensional ultrasonic image; Extracting color labels and a structural prior image of a target view from the preprocessed initial two-dimensional ultrasound image; A cross-view learning network is constructed, and the cross-view learning network is trained based on the preprocessed initial Doppler color image, the color label and the structural prior image of the target view to obtain the pre-trained cross-view learning network.

3. The cross-view blood flow imaging prediction method according to claim 2, characterized in that: The step of extracting the color label and the structural prior image of the target view from the preprocessed initial two-dimensional ultrasound image comprises: Extracting the blood flow frame contour of the preprocessed initial two-dimensional ultrasound image based on a contour detection algorithm; A square ROI region is cut out from the blood flow frame outline, and the ROI region is uniformly cut to a preset size using a bilinear interpolation method to obtain a cut-out image; Convert the cropped image from the RGB color space to the HSV color space to obtain a color-converted image; Using a threshold segmentation method to segment the color-converted image to obtain a color mask; Obtaining the color label by performing an AND operation on the color mask and the basic Doppler color image; The structural prior image of the target view is obtained by performing a negation operation on the color mask and performing an AND operation on the basic Doppler color image, and converting the image into a grayscale image.

4. The cross-view blood flow imaging prediction method according to any one of claims 1 to 3, characterized in that: The receiving of the Doppler color image of the source view and the two-dimensional ultrasonic structure prior image of the target view, and preprocessing the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image, includes: receiving the Doppler color image of the source view and the two-dimensional ultrasonic structure prior image of the target view; Cropping the Doppler color image and the two-dimensional ultrasonic structure prior image to the same size to obtain a cropped Doppler color image and a cropped two-dimensional ultrasonic structure prior image; The cropped Doppler color image and the cropped two-dimensional ultrasonic structure prior image are subjected to pixel normalization processing to obtain the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image.

5. A cross-view blood flow imaging prediction device, characterized in that: include: An image receiving unit is used to receive a Doppler color image of a source view and a two-dimensional ultrasonic structure prior image of a target view, and preprocess the Doppler color image and the two-dimensional ultrasonic structure prior image respectively to obtain a basic Doppler color image and a basic two-dimensional ultrasonic structure prior image; A blood flow prediction unit, used for inputting the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image into a pre-trained cross-view learning network for prediction processing to obtain a blood flow imaging prediction result of a target view; A superposition and fusion unit, used for superimposing and fusing the basic two-dimensional ultrasonic structure prior image and the blood flow imaging prediction result of the target view to obtain a Doppler color ultrasonic image of the target view; The pre-trained cross-view learning network includes an encoder module, a multi-scale cross-attention mechanism module and a decoder module, and the blood flow prediction unit includes: An image input unit, used for inputting the basic Doppler color image and the basic two-dimensional ultrasonic structure prior image into the pre-trained cross-view learning network; A second attention calculation unit is used for dividing the basic Doppler color image and the basic two-dimensional ultrasound structure prior image into non-overlapping image blocks and embedding them into a linear space in the encoder module, and calculating the self-attention of the image blocks; An image block merging unit, used to merge the connected image blocks and calculate the self-attention of the merged image blocks; A feature generation unit, configured to generate a source view feature and a target view structure prior feature based on the self-attention of the image block and the self-attention of the merged image block; An upsampling unit, configured to perform upsampling processing on the keys and values ​​of the high-level semantic features in the source view features in the multi-scale cross-attention mechanism module to obtain the keys and values ​​of the multi-scale low-level semantic features; a third attention calculation unit, configured to use the target view structure prior feature as a query, and perform multi-scale cross attention calculation in combination with the key and value of the multi-scale low-level semantic feature to obtain a multi-scale fused feature map; A feature decoding unit is used to perform upsampling and jump connection processing on the multi-scale fusion feature map through the decoder module to decode the multi-scale fusion feature map to obtain a blood flow imaging prediction result of the target view.

6. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the cross-view blood flow imaging prediction method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the cross-view blood flow imaging prediction method according to any one of claims 1 to 4 is implemented.

Images