Ultrasound image quantization method based on interactive fusion of Transformer

Abstract

The present invention discloses an ultrasound image quantization method using an interactive fusion transformer, the method comprising: inputting the real image into a trained image segmentation model, and outputting an image segmentation prediction mask through the image quantization analysis model; wherein the image segmentation model comprises several U-shaped encoding / decoding modules (including a convolutional neural network, a dual-attention dimensionality reduction transformer network for extracting spatial features and channel features, and a bidirectional interactive fusion unit), and based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result. The present invention enhances the learning ability of global context information by extracting spatial features and channel features through a dual-attention dimensionality reduction transformer network, and interactively learns between the convolutional neural network and the transformer network branches to achieve bidirectional fusion between local features and global context information.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an ultrasound image quantization method using an interactive fusion Transformer. Background Art

[0002] Congenital heart disease is the leading cause of neonatal and infant mortality, with an incidence rate of 8‰ to 1%, ranking first among birth defects. Determining the disease type through imaging equipment is undoubtedly a prerequisite for controlling and treating its progression. Echocardiography, due to its non-invasiveness, portability, low cost, and minimal requirement for infant cooperation, has become the preferred device for assessing cardiac health. Clinically, segmenting various echocardiographic anatomical structures is crucial for physicians to understand various parameters such as volume and diameter. This process often requires physicians to manually delineate the anatomical structures and measure relevant biological parameters, which is time-consuming and labor-intensive, and can result in significant inter-observer variability. Therefore, there is an urgent need to develop an effective, robust, and quantitative method for automated segmentation and measurement to reduce physician workload and inter-observer variability, thereby improving the effectiveness of clinically assisted screening.

[0003] However, current image quantitative analysis methods result in inaccurate quantitative analysis results due to poor segmentation accuracy or neglect of local detail information.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an ultrasound image quantization method based on interactive fusion of Transformer in response to the above-mentioned defects of the prior art, aiming to solve the problem that the image quantization analysis method in the prior art has inaccurate quantitative analysis results due to poor segmentation accuracy or neglect of local detail information.

[0006] The technical solutions adopted by the present invention to solve the problem are as follows:

[0007] In a first aspect, an embodiment of the present invention provides an ultrasound image quantization method using interactive fusion of Transformers, wherein the method includes:

[0008] Collect real images;

[0009] The real image is input into a trained image segmentation model, and an image segmentation prediction mask is output through the image quantization analysis model; wherein the image segmentation model includes a plurality of U-shaped encoding / decoding modules, and the plurality of U-shaped encoding / decoding modules are sequentially cascaded and jump-connected; each of the U-shaped encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network, and a bidirectional interactive fusion unit, and the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features;

[0010] Based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result.

[0011] In one implementation, inputting the real image into a trained image segmentation model and outputting an image segmentation prediction mask through the image quantization analysis model includes:

[0012] Serializing the real image to obtain an image sequence;

[0013] Input the image sequence into the dual-attention dimension reduction Transformer network in the U-type encoding / decoding module, and input the real image into the convolutional neural network in the U-type encoding / decoding module;

[0014] For each U-shaped encoding / decoding module, based on the bidirectional interactive fusion unit, the first feature output by the dual-attention dimensionality reduction Transformer network in each U-shaped encoding / decoding module is fused with the second feature output by the convolutional neural network in each U-shaped encoding / decoding module to obtain a fusion result, and the fusion result is respectively input into the dual-attention dimensionality reduction Transformer network in the next-level U-shaped encoding / decoding module and the convolutional neural network in the U-shaped encoding / decoding module, until all U-shaped encoding / decoding modules are used and the target fusion feature is output;

[0015] The target fusion features are input into the dual-attention dimension reduction Transformer network to obtain the image segmentation prediction mask.

[0016] In one implementation, the bidirectional interactive fusion unit includes a group feature learning unit and a channel compression excitation unit, and the first feature output by the dual-attention dimension reduction Transformer network in each of the U-type encoding / decoding modules is fused with the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules to obtain a fusion result including:

[0017] Inputting the second feature output by the convolutional neural network in each of the U-shaped encoding / decoding modules into the grouping feature learning unit to obtain a third feature, and fusing the third feature with the first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-shaped encoding / decoding modules to obtain a first fusion result;

[0018] The first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-type encoding / decoding modules is input into the channel compression excitation unit to obtain a fourth feature, and the fourth feature is fused with the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules to obtain a second fusion result.

[0019] In one implementation, the fusion results are respectively input into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module and the convolutional neural network in the U-type encoding / decoding module until all U-type encoding / decoding modules are used, and the output target fusion features include:

[0020] The first fusion result is input into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module, and the second fusion result is input into the convolutional neural network in the next-level U-type encoding / decoding module until all U-type encoding / decoding modules are used and the target fusion features are output.

[0021] In one implementation, the dual-attention dimensionality reduction Transformer network includes a multi-head dimensionality reduction dual-attention module and a feedforward network module.

[0022] In one implementation, the multi-head dimensionality reduction dual attention module includes a spatial self-attention module, a normalization layer, and a channel self-attention residual link module.

[0023] In one implementation, the training process of the image segmentation model includes:

[0024] Acquire initial training data, and expand the initial training data to obtain target training data;

[0025] Determining, based on the target training data, a label corresponding to the target training data;

[0026] The target training data is input into the initial image segmentation model to obtain model output data, and the initial image segmentation model is trained based on the model output data and the label. When the training meets the preset conditions, an image segmentation model is obtained.

[0027] In one implementation, performing positioning and quantitative analysis on the image segmentation prediction mask based on a preset positioning method to obtain a quantitative analysis result includes:

[0028] Based on the key point positioning method, the image segmentation prediction mask is positioned to obtain a number of positioning points;

[0029] Calculating the plurality of positioning points to obtain a plurality of inner diameters corresponding one-to-one to a plurality of partial areas in the real image;

[0030] All inner diameters were used as quantitative analysis results.

[0031] In a second aspect, an embodiment of the present invention further provides an ultrasound image quantization device using interactive fusion of Transformers, wherein the device comprises:

[0032] Acquisition module, used to acquire real images;

[0033] A segmentation module, configured to input the real image into a trained image segmentation model and output an image segmentation prediction mask through the image quantization analysis model; wherein the image segmentation model includes a plurality of U-shaped encoding / decoding modules, which are sequentially cascaded and jump-connected; each of the U-shaped encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network, and a bidirectional interactive fusion unit, wherein the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features;

[0034] The positioning and quantitative analysis module is used to perform positioning and quantitative analysis on the image segmentation prediction mask based on a preset positioning method to obtain a quantitative analysis result.

[0035] In a third aspect, an embodiment of the present invention also provides an intelligent terminal comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include an ultrasound image quantization method for executing an interactive fusion Transformer as described in any one of the above.

[0036] In a fourth aspect, an embodiment of the present invention further provides a non-temporary computer-readable storage medium, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the ultrasound image quantization method of the interactive fusion Transformer as described in any one of the above.

[0037] Beneficial effects of the present invention: The embodiment of the present invention first collects a real image; then inputs the real image into a trained image segmentation model, and outputs an image segmentation prediction mask through the image quantization analysis model; wherein, the image segmentation model includes a plurality of U-type encoding / decoding modules, and the plurality of U-type encoding / decoding modules are cascaded and jump-connected in sequence; each of the U-type encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network and a bidirectional interactive fusion unit, and the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features; finally, based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result; it can be seen that in the embodiment of the present invention, the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features to enhance the learning ability of global context information, and the convolutional neural network and the Transformer network branches interact and learn from each other to achieve bidirectional fusion between local features and global context information. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A flow chart of the ultrasound image quantization method using interactive Transformer fusion provided in an embodiment of the present invention.

[0040] Figure 2 A diagram of the interactive fusion Transformer network structure framework for an implementation method provided in an embodiment of the present invention.

[0041] Figure 3 A structural diagram of a group feature learning unit according to an implementation method provided in an embodiment of the present invention.

[0042] Figure 4 A structural design diagram of a DPT according to an implementation method provided by an embodiment of the present invention.

[0043] Figure 5 A structural diagram of a dual self-attention module according to an implementation method provided in an embodiment of the present invention.

[0044] Figure 6 This is a principle block diagram of the ultrasonic image quantization device using interactive fusion Transformer provided in an embodiment of the present invention.

[0045] Figure 7 This is a block diagram of the internal structure of the smart terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention discloses an ultrasound image quantization method using an interactive Transformer fusion. To clarify the objectives, technical solutions, and effects of the present invention, the present invention is further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the present invention and are not intended to limit the present invention.

[0047] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0048] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0049] In the existing technology, the key to achieving clinical auxiliary screening through quantitative analysis is whether the accurate segmentation of anatomical structures can be achieved. In the early work of segmentation algorithms, pixel classification, deformable methods, shape prior models, etc. were mainly used. Such methods achieved good segmentation effects in CT and MRI images from the perspectives of pixel intensity and shape prior. However, echocardiography has inherent limitations such as blurred boundaries, speckle noise, and large differences in anatomical structure shapes. It is not enough to accurately outline the boundary information of anatomical structures based solely on information such as energy and shape prior. Secondly, such traditional methods usually require manual design of targeted feature extraction schemes, which are computationally intensive, have a low scope of applicability, and have limited segmentation effects.

[0050] In contrast, deep learning algorithms, with their powerful feature learning capabilities, can autonomously learn complex features and complete end-to-end segmentation tasks. Since the UNet algorithm was first introduced for medical image segmentation in 2015, this type of U-shaped convolutional neural network and its variants have been continuously applied to various disease segmentation tasks, achieving high segmentation accuracy. However, these methods often neglect the representation of global contextual information in the network design, which is crucial for accurate structural segmentation.

[0051] Based on the consideration of extracting global context information, some scholars have turned their attention to the transformer model, which has been very successful in natural language processing. This model can learn large-scale dependencies between input blocks and realize the extraction of global context information. Although the use of the transformer model can realize the consideration of the global context information of the data, this type of method does not take into account local detail information. In order to effectively utilize local and global information, algorithms that combine CNN and transformer continue to emerge and have been applied in the field of medical image segmentation, such as UNet Transformer, TransUnet, Multi-Compound Transformer, etc. However, most of these methods consider replacing convolutional layers or stacking the features extracted by both to achieve the connection between local and global information, which makes it difficult to maximize the learning ability of the two models.

[0052] In order to solve the problems of the prior art, this embodiment provides an ultrasound image quantization method of interactive fusion Transformer, which extracts spatial features and channel features through a dual-attention dimensionality reduction Transformer network to enhance the learning ability of global context information, and interactive learning between the convolutional neural network and the Transformer network branches to achieve bidirectional fusion between local features and global context information. In specific implementation, a real image is first collected; then the real image is input into a trained image segmentation model, and an image segmentation prediction mask is output through the image quantization analysis model; wherein, the image segmentation model includes several U-type encoding / decoding modules, and the several U-type encoding / decoding modules are cascaded and jump-connected in sequence; each of the U-type encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network and a bidirectional interactive fusion unit, and the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features; finally, based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result.

[0053] Exemplary Methods

[0054] This embodiment provides an ultrasound image quantization method that interactively integrates Transformers, which can be applied to artificial intelligence smart terminals. Figure 1 As shown, the method includes:

[0055] Step S100: collecting real images;

[0056] In this embodiment, the real image is two types of echocardiographic data of different sections (pediatric parasternal cardiac short-axis image and pediatric four-chamber echocardiogram), such as the parasternal cardiac short-axis image, and the real image can be obtained by acquisition.

[0057] After getting the real image, you can execute Figure 1 The following steps are shown: S200, inputting the real image into the trained image segmentation model, and outputting the image segmentation prediction mask through the image quantization analysis model; wherein the image segmentation model includes a plurality of U-shaped encoding / decoding modules, and the plurality of U-shaped encoding / decoding modules are sequentially cascaded and jump-connected; each of the U-shaped encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network and a bidirectional interactive fusion unit, and the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features;

[0058] Specifically, the image segmentation model has been pre-trained and is used for quantitative analysis of infant echocardiography by establishing interactive learning between convolutional branches and Transformer branches. In this embodiment, the image segmentation model includes seven U-shaped encoding / decoding modules, and the number used in practice is not limited. Each U-shaped encoding / decoding module represents a different feature learning stage with a similar structure. The seven U-shaped encoding / decoding modules are cascaded in sequence. Each U-shaped encoding / decoding module includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network, and a bidirectional interactive fusion unit. In this way, the output features of the convolutional neural network and the features output by the dual-attention dimensionality reduction Transformer network are interactively learned through the bidirectional interactive fusion unit, achieving bidirectional fusion between local features and global context information. The dual-attention dimensionality reduction Transformer network uses a dual-attention pyramid dimensionality reduction module to model large-scale dependencies in space and channels. In addition, because the dual-attention dimensionality reduction Transformer network extracts spatial and channel features, it enables the establishment of large-scale dependencies and enhances the learning ability of global context information.

[0059] In one implementation, step S200 includes the following steps:

[0060] S201, serializing the real image to obtain an image sequence;

[0061] S202: Input the image sequence into the dual-attention dimension reduction Transformer network in the U-type encoding / decoding module, and input the real image into the convolutional neural network in the U-type encoding / decoding module;

[0062] S203. For each U-shaped encoding / decoding module, based on the bidirectional interactive fusion unit, fuse the first feature output by the dual-attention dimensionality reduction Transformer network in each U-shaped encoding / decoding module with the second feature output by the convolutional neural network in each U-shaped encoding / decoding module to obtain a fusion result. The fusion result is respectively input into the dual-attention dimensionality reduction Transformer network in the next-level U-shaped encoding / decoding module and the convolutional neural network in the U-shaped encoding / decoding module, until all U-shaped encoding / decoding modules are used and the target fusion feature is output;

[0063] S204: Input the target fusion features into a dual-attention dimension reduction Transformer network to obtain an image segmentation prediction mask.

[0064] Specifically, the real image is serialized to obtain an image sequence, the image sequence is used as the initial input of the dual-attention dimension reduction Transformer network, and the real image is used as the input of the convolutional neural network, as shown in Figure 2As shown, the first line is a dual-attention dimension reduction Transformer network (DPT branch). The second line is a bidirectional interactive fusion unit. The third line is a convolutional neural network (CNN branch). Xi is the output feature of the DPT block of the i-th layer. Yi is the output feature of the CNN block of the i-th layer. For each of the U-type encoding / decoding modules, based on the bidirectional interactive fusion unit, the first feature output by the dual-attention dimension reduction Transformer network in each of the U-type encoding / decoding modules is fused with the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules to obtain a fusion result, and the fusion result is respectively input into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module and the convolutional neural network in the U-type encoding / decoding module until all U-type encoding / decoding modules are used and the target fusion feature is output. That is, in this embodiment, after passing through 7 U-type encoding / decoding modules, the target fusion feature will eventually be output and the target fusion feature will be input into the dual-attention dimension reduction Transformer network. That is, the output of the last layer of dual-attention dimension reduction Transformer network that fuses local information is selected as the final segmentation prediction map, and the final image segmentation prediction mask is generated through the Softmax function. It's worth noting that the final layer's dual-attention dimensionality reduction Transformer network is separate from the one in the U-shaped encoding / decoding module; it's a separate layer with the same structure. It performs a bidirectional interactive fusion unit operation on the features extracted at each layer, integrating local and global features by setting up two opposing fusion paths.

[0065] In step S203, the bidirectional interactive fusion unit includes a group feature learning unit and a channel compression excitation unit, and the first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-type encoding / decoding modules is fused with the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules to obtain a fusion result, including the following steps: inputting the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules into the group feature learning unit to obtain a third feature, and fusing the third feature with the first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-type encoding / decoding modules to obtain a first fusion result; inputting the first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-type encoding / decoding modules into the channel compression excitation unit to obtain a fourth feature, and fusing the fourth feature with the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules to obtain a second fusion result.

[0066] Specifically, in order to effectively combine the global and local features extracted at each stage of the dual-branch network, a bidirectional interactive fusion unit is used to achieve interactive fusion of different features. This unit contains two directional feature enhancement modules, namely, a group feature learning unit for fusing local features to global features, and a channel compression excitation unit in the opposite direction, such as Figure 2 As shown. Group feature learning unit (GFL), such as Figure 3 As shown, SPA is the spatial attention module. CHA is the channel attention module. i is the feature of the i-th layer of the CNN branch. t is the initial grouping feature. d is the expansion rate. To enhance group features. is the enhanced feature of the i-th layer obtained after processing by the group feature learning unit. First, we perform the initial local feature Y i Group t times according to the number of channels to obtain mutually independent sub-features P t Then for each sub-feature P t Perform spatial pyramid pooling operation, which pools P according to different (expansion rate) void rate t Divided into 4 branches with different receptive fields (n is the number of branches) to capture multi-scale information. SPA blocks are then executed on each branch to explicitly model the interdependence between feature channels. Based on this, the spatial attention module is executed to train the network to focus on the features of the target area, enhance the expression of detailed information, and suppress the information of irrelevant areas, resulting in See Figure 4 Finally, the original features are fused with the features of different branches through cascade operations, and the channel dimension is reduced using 1×1 convolution to obtain For each sub-feature Aggregate and obtain through channel confusion operator Realize information exchange among various sub-features along the channel direction.

[0067] Channel compression excitation unit: This unit is through the input feature X i Perform global average pooling to achieve contextual information modeling. Then, the feature map information of each channel is fused according to the two fully connected layers to achieve cross-channel information interaction. And generate a weight map through the Sigmoid function and combine it with the input feature X. i The enhanced global features are obtained by channel-by-channel multiplication and compare it with Y i Add them together and use them as the input of the next layer of CNN.

[0068] For the first U-shaped encoding / decoding module and the last U-shaped encoding / decoding module, the channel compression excitation unit in the bidirectional interactive fusion unit in the first U-shaped encoding / decoding module and the last U-shaped encoding / decoding module is set to 0; that is, the bidirectional interactive fusion unit in the first U-shaped encoding / decoding module and the last U-shaped encoding / decoding module is a unidirectional bridge fusion unit. For the middle 5 U-shaped encoding / decoding modules, the second feature Y output by the convolutional neural network in the current level U-shaped encoding / decoding module is i Input to the group feature learning unit to obtain the third feature, and combine the third feature with the first feature F output by the dual attention dimension reduction Transformer network in the current level U-type encoding / decoding module i Fusion is performed to obtain the first fusion result of the current level, in preparation for the input of the next level; the first feature F output by the dual attention dimension reduction Transformer network in the U-shaped encoding / decoding module of the current level is converted into i Input to the channel compression excitation unit to obtain the fourth feature, and combine the fourth feature with the second feature Y output by the convolutional neural network in the current level U-type encoding / decoding module i Perform fusion to obtain the second fusion result of the current level and prepare for the input of the next level.

[0069] In step S203, the fusion results are respectively inputted into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module and the convolutional neural network in the U-type encoding / decoding module until all U-type encoding / decoding modules are used, and the target fusion feature is output, including the following steps: inputting the first fusion result into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module, and inputting the second fusion result into the convolutional neural network in the next-level U-type encoding / decoding module until all U-type encoding / decoding modules are used, and outputting the target fusion feature.

[0070] Specifically, the first fusion result of the current level is input into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module, and the second fusion result of the current level is input into the convolutional neural network in the next-level U-type encoding / decoding module, until all 5 U-type encoding / decoding modules are used and the target fusion features are output.

[0071] In one implementation, the dual-attention dimensionality reduction Transformer network includes a multi-head dimensionality reduction dual-attention module and a feedforward network module.

[0072] Specifically, according to the dual-attention dimensionality reduction Transformer network in the U-shaped encoding / decoding module at different levels (that is, the Transformer layer at different stages), multi-scale global context information extraction can be achieved. The dual-attention dimensionality reduction Transformer network uses four Transforme encoding / decoding blocks as the skeleton and realizes the transmission of hidden layer information through jump connections. Since the present invention is to achieve multi-scale structural design, the construction of each Transforme encoding / decoding block needs to consider the change of feature size. Similar to the network design mode of CNN, each encoding block or decoding block can be regarded as a different feature learning stage with similar structure. Therefore, this study designs the structure of the DPT branch by describing the structure of a single encoding block. Figure 4 As shown, F i Serialized data of input features for layer i. i is the output feature of the i-th layer processed by the Transformer block.

[0073] The Transforme encoding / decoding block used in the present invention includes an embedding layer and a multi-head dimensionality reduction dual attention block (DPT layer), such as Figure 4 As shown, where F i Serialized data of input features for layer i. i is the output feature of the i-th layer after being processed by the Transformer block. The specific description is as follows:

[0074] Block embedding: Assume that the input feature of the i-th layer is X i-1 , The block size of the i-th layer is P i We first divide the input features into N i After linear projection, the 2D embedded feature block can be converted into a 1D embedded image sequence with a sequence shape of N i ×C i , where i = 1, 2, 3, 4.

[0075] In one implementation, the multi-head dimensionality reduction dual attention module includes a spatial self-attention module, a normalization layer, and a channel self-attention residual link module.

[0076] Specifically, multi-head dimensionality reduction dual attention block: In order to make full use of remote context information, the present invention connects a channel self-attention module in series on the basis of the original spatial self-attention module, such as Figure 5 As shown, where SR(·) is the spatial size reduction function. i 、W i and C i is the length, width, and number of channels of the input data of layer i. r is the dimensionality reduction rate.

[0077] The specific implementation is as follows:

[0078] Step 1: Assume that the feature sequence F of the i-th layer is processed by block embedding i As the input of the original spatial self-attention module. After the spatial self-attention module, global spatial features are learned, capturing large-scale dependencies in the spatial dimension. Then, residual links are used to aggregate the global context features into the original features to obtain S.

[0079]

[0080]

[0081] Where, SR(·) is the spatial size reduction function, are three learnable parameter matrices, and T is the transposed symbol. M i Indicates that the multi-head self-attention block in layer i contains M i independent self-attention layers, i.e., the number of heads. Norm(·) is the layer normalization function, Reshape(F i ,r) is the spatial dimensionality reduction operator of the feature sequence of layer i, and r is the dimensionality reduction rate. Assume that the input feature After spatial dimensionality reduction, the output features is a linear projection, which is used to change the number of input feature channels.

[0082] Step 2: The feature S obtained by the spatial self-attention residual link is layer-normalized to obtain L, which is then used as the input of the channel self-attention residual link module. A self-attention mechanism similar to Step 1 is used to capture the channel dependency between any two channel mappings, and the output G of the entire self-attention block is obtained. According to the matrix multiplication principle, q and k need to be reduced in dimension here, as shown in the following example: Figure 5 shown.

[0083] L=Norm(S) (3)

[0084]

[0085] in, are three learnable parameter matrices,

[0086] Multi-head Dimensionality Reduction Dual Self-Attention Module (MDA): MDA projects the same q, k, v vectors (query, key, value vectors) into different subspaces of the original high-dimensional space and then performs dual self-attention calculations. The output of each subspace is then cascaded to obtain the final global information. Its specific representation is as follows:

[0087]

[0088] in is a learnable linear projection matrix.

[0089] Based on the above work, the final output of the DPT layer is X i :

[0090] X i =FFN(F i +MDA(F i ))+(F i +MDA(F i )) (6)

[0091] Among them, FFN() is a feedforward network, which contains two linear transformations and a ReLU activation function.

[0092] In one implementation, the training process of the image segmentation model includes the following steps: obtaining initial training data, and expanding the initial training data to obtain target training data; determining a label corresponding to the target training data based on the target training data; inputting the target training data into the initial image segmentation model to obtain model output data, and training the initial image segmentation model based on the model output data and the label, and obtaining an image segmentation model when the training meets preset conditions.

[0093] Specifically, initial training data was obtained from two different types of echocardiographic data (pediatric parasternal short-axis echocardiography and pediatric four-chamber echocardiography) to verify segmentation performance and measure the internal diameters of anatomical structures in pediatric parasternal short-axis echocardiography. Both datasets were collected from Shenzhen Children's Hospital. The four-chamber image dataset contained 127 four-chamber video sequences, 100 of which were used for training and 27 for testing. Through video sequence-to-image conversion, a total of 3,654 training data sets and 831 testing data sets were included. The parasternal short-axis echocardiography dataset collected 50 parasternal short-axis echocardiography video sequences, 40 of which were used as training data and 10 as testing data. After video sequence conversion, a total of 1,273 training data sets and 350 testing data sets were included. Due to the small size of this dataset, it is not sufficient to train the network model well and is prone to overfitting. Therefore, this study enhanced the initial training data (pediatric dataset) by modifying image brightness, contrast, and rotation angle, increasing the target training data (data volume) to 8911 images. Secondly, we selected 3-4 ultrasound physicians to delineate structures such as the left atrium, left ventricle, aortic annulus, and right ventricular outflow tract in different datasets and create labels for segmentation to ensure the quality of the segmented datasets.

[0094] Experimental Environment: This experiment was conducted using the PyTorch framework on an Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, an NVIDIA Titan Xp GPU, and 64GB of RAM. All images were resized to 256*256 before training and testing. Mini-batch stochastic gradient descent was used as the optimization method for 50 epochs. The default conditions were to train for 50 epochs, with an initial learning rate of 1e-3 and a momentum of 0.99.

[0095] Method evaluation indicators: The experiment of the present invention is divided into two sub-tasks: segmentation and quantization. Therefore, there are two different types of evaluation indicators to judge the performance of the two tasks respectively.

[0096] The segmentation task is evaluated using the Dice coefficient (Dice), Jaccard similarity coefficient (Jaccard), sensitivity (Sensitivity), and precision (Precision) indicators. The calculation method is as follows:

[0097]

[0098]

[0099]

[0100]

[0101] TP, FP, and FN are the number of true positive, false positive, and false negative pixels, respectively.

[0102] The evaluation indicators of the quantization task are composed of the mean absolute error (MAE) and the root mean square error (RMSE). The calculation method is as follows:

[0103]

[0104]

[0105] X and Y are two different sets of data.

[0106] After obtaining the image segmentation prediction mask, you can perform the following Figure 1 The following steps are shown: S300, based on a preset positioning method, positioning and quantitative analysis are performed on the image segmentation prediction mask to obtain a quantitative analysis result.

[0107] In this embodiment, based on a preset positioning method, the tricuspid valve heel and the pulmonary valve heel are positioned to determine the length and direction of the inner diameter, thereby achieving quantitative analysis of the anatomical structure.

[0108] Step S300 includes the following steps:

[0109] S301, positioning the image segmentation prediction mask based on a key point positioning method to obtain a number of positioning points;

[0110] S302, calculating the plurality of positioning points to obtain a plurality of inner diameters corresponding to a plurality of partial regions in the real image;

[0111] S303. Take all inner diameters as quantitative analysis results.

[0112] In this embodiment, the preset positioning method is the key point positioning method. The parasternal heart short axis image of children is used as the training data set. It is intended to achieve the segmentation of the right ventricular outflow tract and aortic valve annulus through bidirectional fusion Transformer, and to achieve the measurement of the aortic diameter, right ventricular outflow tract diameter, and pulmonary artery diameter through the key point positioning method. The specific method is as follows:

[0113] Step 1: Locate the highest point a in the AO area, and then find the point b closest to the AO in the RVOT area.

[0114] Step 2: On the right side of point b in the RVOT region, the y-axis maximum point corresponding to a certain coordinate point is point d, and the x-axis maximum point is point e.

[0115] Step 3: On the left side of point b in the RVOT region, the point with the maximum y-axis value corresponding to a coordinate point is point f. Point f and point b define a line segment, and the point farthest from point fb in the area below fb is point g.

[0116] Step 4: Locate the points m and o closest to g and d in the AO area as the base points for calculating the inner diameter of the aorta.

[0117] Step 5: Finally, the point where the line perpendicular to the midpoint of mo intersects with the boundary of AO is located as point p and point u.

[0118] Step 6: Locate the point where the line perpendicular to the midpoint of mo intersects the boundary of the RVOT region as v and z.

[0119] Pulmonary artery diameter: the distance between de.

[0120] Inner diameter of aorta: The distance between line segments pu is the inner diameter length of AO.

[0121] Right ventricular outflow tract internal diameter: Line segment vz is the right ventricular outflow tract internal diameter, and the internal diameter length is calculated according to the distance formula.

[0122] Anatomical segmentation and measurement in echocardiography are key steps in achieving quantitative analysis of congenital heart disease. However, inherent limitations of echocardiography, such as blurred boundaries, speckle noise, and large variations in anatomical shape, complicate quantitative analysis. Current methods address this challenge by constructing CNN algorithms, but these algorithms lack consideration of global context and large-scale dependencies. Therefore, to improve the performance of quantitative analysis of pediatric echocardiography, this paper proposes an interactive fusion Transformer network. By constructing interactive learning between convolutional and Transformer branches, this network achieves bidirectional fusion of local features and global context. First, a dual-attention pyramid dimensionality reduction Transformer branch is implemented to establish large-scale dependencies from both spatial and channel perspectives, enhancing the learning of global context. Second, a bidirectional interactive fusion unit is implemented to achieve interactive fusion between local and global features, maximizing the preservation of local and global features and refining the segmentation results. Finally, anatomical parameter measurement is achieved by simulating clinical parameter measurement methods through key point localization. The embodiment of the present invention selects the parasternal aortic root short-axis image of infants and young children as experimental data to achieve the segmentation of the right ventricular outflow tract and the aortic valve annulus and the measurement of the corresponding inner diameter, verifying that the algorithm has the potential to serve as a clinical auxiliary tool.

[0123] Exemplary devices

[0124] like Figure 6As shown in , an embodiment of the present invention provides an ultrasound image quantization device interactively fused with Transformer, the device comprising an acquisition module 401, a segmentation module 402, and a positioning and quantitative analysis module 403, wherein:

[0125] Acquisition module 401, used for acquiring real images;

[0126] Segmentation module 402, configured to input the real image into a trained image segmentation model and output an image segmentation prediction mask through the image quantization analysis model; wherein the image segmentation model includes a plurality of U-shaped encoding / decoding modules, which are sequentially cascaded and jump-connected; each of the U-shaped encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network, and a bidirectional interactive fusion unit, wherein the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features;

[0127] The positioning and quantitative analysis module 403 is configured to perform positioning and quantitative analysis on the image segmentation prediction mask based on a preset positioning method to obtain a quantitative analysis result.

[0128] Based on the above embodiment, the present invention also provides an intelligent terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The intelligent terminal includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected via a system bus. The processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an ultrasound image quantification method of an interactive fusion transformer is implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the intelligent terminal is pre-set inside the intelligent terminal to detect the operating temperature of the internal device.

[0129] Those skilled in the art will understand that Figure 7 The schematic diagram is only a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the smart terminal to which the solution of the present invention is applied. The specific smart terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0130] In one embodiment, a smart terminal is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations:

[0131] Collect real images;

[0132] The real image is input into a trained image segmentation model, and an image segmentation prediction mask is output through the image quantization analysis model; wherein the image segmentation model includes a plurality of U-shaped encoding / decoding modules, and the plurality of U-shaped encoding / decoding modules are sequentially cascaded and jump-connected; each of the U-shaped encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network, and a bidirectional interactive fusion unit, and the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features;

[0133] Based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result.

[0134] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0135] In summary, the present invention discloses an ultrasound image quantization method of interactive fusion Transformer, which includes: inputting the real image into a trained image segmentation model, and outputting an image segmentation prediction mask through the image quantization analysis model; wherein, the image segmentation model includes several U-shaped encoding / decoding modules (including a convolutional neural network, a dual-attention dimensionality reduction Transformer network for extracting spatial features and channel features, and a bidirectional interactive fusion unit), and based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result. The present invention enhances the learning ability of global context information by extracting spatial features and channel features through a dual-attention dimensionality reduction Transformer network, and interactively learns between the convolutional neural network and the Transformer network branches to achieve bidirectional fusion between local features and global context information.

[0136] Based on the above embodiments, the present invention discloses an ultrasound image quantization method using interactive fusion Transformer. It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, it can be improved or transformed according to the above description. All these improvements and transformations should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An ultrasound image quantization method using interactive fusion Transformer, characterized in that: The method comprises: Collect real images; The real image is input into a trained image segmentation model, and an image segmentation prediction mask is output by the image segmentation model; wherein the image segmentation model includes a plurality of U-type encoding / decoding modules, and the plurality of U-type encoding / decoding modules are sequentially cascaded and jump-connected; each of the U-type encoding / decoding modules includes a convolutional neural network, a dual-attention dimensionality reduction Transformer network, and a bidirectional interactive fusion unit, and the dual-attention dimensionality reduction Transformer network is used to extract spatial features and channel features; for each U-type encoding / decoding module, based on the bidirectional interactive fusion unit, the first feature output by the dual-attention dimensionality reduction Transformer network in each U-type encoding / decoding module is fused with the second feature output by the convolutional neural network in each U-type encoding / decoding module to obtain a fusion result, and the fusion result is respectively input into the dual-attention dimensionality reduction Transformer network in the next-level U-type encoding / decoding module and the convolutional neural network in the U-type encoding / decoding module until all U-type encoding / decoding modules are used; Based on a preset positioning method, the image segmentation prediction mask is positioned and quantitatively analyzed to obtain a quantitative analysis result.

2. The ultrasound image quantization method of interactive fusion Transformer according to claim 1 is characterized in that: Inputting the real image into a trained image segmentation model, and outputting an image segmentation prediction mask through the image segmentation model includes: Serializing the real image to obtain an image sequence; Input the image sequence into the dual-attention dimension reduction Transformer network in the U-shaped encoding / decoding module, input the real image into the convolutional neural network in the U-shaped encoding / decoding module, and output the target fusion feature; The target fusion features are input into the dual-attention dimension reduction Transformer network to obtain the image segmentation prediction mask.

3. The ultrasound image quantization method of interactive fusion Transformer according to claim 2 is characterized in that: The bidirectional interactive fusion unit includes a group feature learning unit and a channel compression excitation unit, and the first feature output by the dual-attention dimension reduction Transformer network in each of the U-shaped encoding / decoding modules is fused with the second feature output by the convolutional neural network in each of the U-shaped encoding / decoding modules to obtain a fusion result including: Inputting the second feature output by the convolutional neural network in each of the U-shaped encoding / decoding modules into the grouping feature learning unit to obtain a third feature, and fusing the third feature with the first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-shaped encoding / decoding modules to obtain a first fusion result; The first feature output by the dual-attention dimensionality reduction Transformer network in each of the U-type encoding / decoding modules is input into the channel compression excitation unit to obtain a fourth feature, and the fourth feature is fused with the second feature output by the convolutional neural network in each of the U-type encoding / decoding modules to obtain a second fusion result.

4. The ultrasound image quantization method of interactive fusion Transformer according to claim 3 is characterized in that: The fusion results are respectively input into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module and the convolutional neural network in the U-type encoding / decoding module until all U-type encoding / decoding modules are used, and the output target fusion features include: The first fusion result is input into the dual-attention dimension reduction Transformer network in the next-level U-type encoding / decoding module, and the second fusion result is input into the convolutional neural network in the next-level U-type encoding / decoding module until all U-type encoding / decoding modules are used and the target fusion features are output.

5. The ultrasound image quantization method of interactive fusion Transformer according to claim 2, characterized in that: The dual-attention dimensionality reduction Transformer network includes a multi-head dimensionality reduction dual-attention module and a feedforward network module.

6. The ultrasound image quantization method of interactive fusion Transformer according to claim 5, characterized in that: The multi-head dimensionality reduction dual attention module includes a spatial self-attention module, a normalization layer and a channel self-attention residual link module.

7. The ultrasound image quantization method of interactive fusion Transformer according to claim 1, characterized in that: The training process of the image segmentation model includes: Acquire initial training data, and expand the initial training data to obtain target training data; Determining, based on the target training data, a label corresponding to the target training data; The target training data is input into the initial image segmentation model to obtain model output data, and the initial image segmentation model is trained based on the model output data and the label. When the training meets the preset conditions, an image segmentation model is obtained.

8. The ultrasound image quantization method of interactive fusion Transformer according to claim 1, characterized in that: The image segmentation prediction mask is positioned and quantitatively analyzed based on a preset positioning method to obtain a quantitative analysis result, including: Based on the key point positioning method, the image segmentation prediction mask is positioned to obtain a number of positioning points; Calculating the plurality of positioning points to obtain a plurality of inner diameters corresponding one-to-one to a plurality of partial areas in the real image; All inner diameters were used as quantitative analysis results.

9. An intelligent terminal, characterized in that: The invention comprises a memory, a processor, and an ultrasound image quantization program of an interactive fusion transformer stored in the memory and executable on the processor. When the processor executes the ultrasound image quantization program of the interactive fusion transformer, the steps of the ultrasound image quantization method of the interactive fusion transformer as described in any one of claims 1 to 8 are implemented.

10. A non-transitory computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 8.

Images