Ultrasonic space composite imaging method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of ultrasonic imaging, in particular to an ultrasonic space composite imaging method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring at least two groups of target ultrasonic image data; performing channel splicing on the at least two groups of target ultrasonic image data to obtain target multi-channel image data; inputting the target multi-channel image data into an image processing module to obtain a target space composite image; wherein the image processing module comprises a spatial composite network, in the spatial composite network, weight distribution is carried out on a feature map corresponding to the target multi-channel image data in at least one dimension, and the feature map corresponding to the target multi-channel image data is fused based on the distributed weight to obtain fused image information. According to the method, automatic attention mechanism weight distribution is introduced into ultrasonic space composite imaging, the signal-to-noise ratio can be automatically improved according to the condition of the ultrasonic image, and important detail features are reserved.

Description

Technical Field

[0001] The present invention relates to the field of ultrasonic imaging technology, and more particularly to an ultrasonic spatial compound imaging method and device, electronic equipment, storage medium, and computer program product. Background Art

[0002] Ultrasonic spatial compound imaging is a technology that uses sound beams at different angles to explore the target object and integrates the ultrasonic signals reflected by different sound beams into an image. Ultrasonic spatial compound imaging technology can more clearly display interfaces that are not parallel to the probe surface, while reducing noise and speckle artifacts, and can to a certain extent display structures that are obscured by acoustic shadows or in the area of ​​acoustic attenuation. Existing ultrasonic spatial compound imaging technology synthesizes an image by weighted summation of the pixel points in the image data corresponding to the ultrasonic echo signals obtained by scanning at different angles. However, the existing ultrasonic spatial compound imaging algorithm has at least the following defects: it uses preset fixed weights to synthesize the pixel points of the image data corresponding to different angles. The fixed weight coefficient will cause the characteristics of the scanned target object to be unable to adapt to the image content, and the effect will be rigid, which may cause the diagnostic signs such as the side acoustic shadow to be weakened or even disappear. Summary of the Invention

[0003] The present invention is proposed in view of the above problems. According to one aspect of the present invention, there is provided an ultrasonic spatial compound imaging method, comprising: acquiring at least two sets of target ultrasonic image data, the at least two sets of target ultrasonic image data being obtained by performing echo signal processing on at least two sets of target ultrasonic echo signals, the at least two sets of target ultrasonic echo signals each corresponding to a different acoustic beam incident direction; performing channel splicing on the at least two sets of target ultrasonic image data to acquire target multi-channel image data, wherein the at least two sets of target ultrasonic image data each have a separate channel; inputting the target multi-channel image data into an image processing module to acquire a target spatial compound image; wherein the image processing module includes a spatial compound network, in which feature maps corresponding to the target multi-channel image data are weighted in at least one dimension, and the feature maps corresponding to the target multi-channel image data are fused based on the assigned weights to obtain fused image information, and the fused image information is used to acquire the target spatial compound image.

[0004] Further, the at least one dimension includes a channel dimension and / or a spatial dimension.

[0005] Furthermore, the target multi-channel image data is input into the image processing module to obtain the target spatial composite image, including: performing the following operations through the spatial composite network: extracting features from the target multi-channel image data to obtain the original feature map of the target multi-channel image data; assigning channel attention weights and / or spatial attention weights to the original feature map; performing channel attention operations and / or spatial attention operations on the original feature map based on the assigned channel attention weights and / or spatial attention weights to obtain a calculated feature map; generating fused image information based on the calculated feature map; wherein the fused image information is the target spatial composite image, or, inputting the target multi-channel image data into the image processing module to obtain the target spatial composite image, also includes: generating the target spatial composite image based on the fused image information.

[0006] Furthermore, channel attention weights are allocated to the original feature map, including: performing a first pooling operation along the spatial dimension on the original feature map to obtain a first feature map; inputting the first feature map into a multi-layer perceptron to obtain a second feature map output by the multi-layer perceptron; normalizing the eigenvalues ​​in the second feature map to obtain a normalized third feature map, wherein the eigenvalues ​​of the third feature map under each channel are channel attention weights corresponding one-to-one to each channel of the original feature map; performing a channel attention operation on the original feature map based on the allocated channel attention weights to obtain an operated feature map, including: multiplying the channel attention weight corresponding to each channel in the original feature map by the corresponding channel of the original feature map to obtain an operated feature map.

[0007] Furthermore, spatial attention weights are assigned to the original feature map, including: performing a second pooling operation along the channel dimension on the original feature map to obtain a fourth feature map; performing convolution on the fourth feature map, and normalizing the eigenvalues ​​in the feature map obtained by the convolution to obtain a normalized fifth feature map, wherein the eigenvalue corresponding to each of the multiple first feature map areas in the fifth feature map represents the spatial attention weight of the corresponding second feature map area in the original feature map, and the position of the first feature map area in the fifth feature map is consistent with the position of the corresponding second feature map area in the original feature map; performing a spatial attention operation on the original feature map based on the assigned spatial attention weight to obtain an operated feature map, including: multiplying the spatial attention weight corresponding to each second feature map area in the original feature map by the corresponding second feature map area of ​​the original feature map to obtain a operated feature map.

[0008] Furthermore, the original feature map is assigned channel attention weights and spatial attention weights, including: performing a first pooling operation along the spatial dimension on the original feature map to obtain a sixth feature map; normalizing the eigenvalues ​​in the sixth feature map to obtain a normalized seventh feature map, wherein the eigenvalues ​​of the seventh feature map under each channel are the channel attention weights corresponding to each channel of the original feature map; performing a second pooling operation along the channel dimension on the original feature map to obtain an eighth feature map; performing convolution on the eighth feature map, and normalizing the eigenvalues ​​in the feature map obtained by the convolution to obtain a normalized ninth feature map, wherein each of the plurality of third feature map regions in the ninth feature map is a channel attention weight corresponding to each channel of the original feature map. The eigenvalue corresponding to the third feature map area represents the spatial attention weight of the corresponding fourth feature map area in the original feature map, and the position of the third feature map area in the ninth feature map is consistent with the position of the corresponding fourth feature map area in the original feature map; based on the allocated channel attention weight and spatial attention weight, channel attention operation and spatial attention operation are performed on the original feature map to obtain a calculated feature map, including: multiplying the channel attention weight corresponding to each channel in the original feature map by the corresponding channel of the original feature map, and multiplying the spatial attention weight corresponding to each fourth feature map area in the original feature map by the corresponding fourth feature map area of ​​the original feature map to obtain a calculated feature map.

[0009] Furthermore, the spatial compound network is trained in the following manner: obtaining at least two groups of sample ultrasonic image data and corresponding annotated spatial compound images, wherein the at least two groups of sample ultrasonic image data are obtained by performing echo signal processing on at least two groups of sample ultrasonic echo signals, and the acoustic beam incident directions corresponding to the at least two groups of sample ultrasonic echo signals are different; channel stitching is performed on the at least two groups of sample ultrasonic image data to obtain sample multi-channel image data, wherein the at least two groups of sample ultrasonic image data each have a separate channel; the sample multi-channel image data is input into the image processing module to obtain a predicted spatial compound image; and the loss value of the spatial compound network in the image processing module is calculated based on the annotated spatial compound image and the predicted spatial compound image to train the spatial compound network.

[0010] Furthermore, the loss value of the spatial composite network is the pixel distance between the predicted spatial composite image and the annotated spatial composite image, where the pixel distance is calculated by the mean square error.

[0011] Furthermore, the input end of the image processing module is connected in series to the output end of the image preprocessing module, wherein the image preprocessing module is used to perform channel stitching on at least two groups of target ultrasound images to obtain target multi-channel image data. The image preprocessing module has a specific number of input ends. When the number of groups of target ultrasound image data input into the image preprocessing module is less than a specific number, the input data corresponding to the input end without target ultrasound image data input is set to 0.

[0012] Furthermore, channel stitching is performed on at least two groups of target ultrasonic image data to obtain target multi-channel image data, including: if the number of groups of the at least two groups of target ultrasonic image data is greater than or equal to 3, channel stitching is performed on the at least two groups of target ultrasonic image data by setting the i-th group of target ultrasonic image data in the middle channel to obtain target multi-channel image data; wherein, i=floor(n / 2)+1, floor(n / 2) represents rounding down n / 2, n is the number of groups of the at least two groups of target ultrasonic image data, n is an odd number, and the sound beam incident direction of the target ultrasonic echo signal corresponding to the im-th group of target ultrasonic image data and the sound beam incident direction of the target ultrasonic echo signal corresponding to the i+m-th group of target ultrasonic image data satisfy angular symmetry with respect to the sound beam incident direction of the target ultrasonic echo signal corresponding to the i-th group of target ultrasonic image data, and m=1,2,...,floor(n / 2).

[0013] According to the second aspect of the present invention, an ultrasonic spatial compound imaging device is also provided, including: an acquisition module, which is used to acquire at least two groups of target ultrasonic image data, and the at least two groups of target ultrasonic image data are obtained by performing echo signal processing on at least two groups of target ultrasonic echo signals, and the sound beam incident directions corresponding to the at least two groups of target ultrasonic echo signals are different; a splicing module, which is used to perform channel splicing on the at least two groups of target ultrasonic image data to obtain target multi-channel image data, wherein the at least two groups of target ultrasonic image data each have a separate channel; an input module, which is used to input the target multi-channel image data into the image processing module to obtain a target spatial compound image; wherein the image processing module includes a spatial compound network, in which the feature map corresponding to the target multi-channel image data is weighted in at least one dimension, and the feature map corresponding to the target multi-channel image data is fused based on the assigned weight to obtain fused image information, and the fused image information is used to obtain the target spatial compound image.

[0014] According to a third aspect of the present invention, an electronic device is further provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the above-mentioned ultrasonic spatial compound imaging method when executed.

[0015] According to a fourth aspect of the present invention, a storage medium is further provided, on which program instructions are stored. The program instructions are used to execute the above-mentioned ultrasonic spatial compound imaging method when running.

[0016] According to a fifth aspect of the present invention, a computer program product is further provided, comprising computer program instructions, which are used to execute the ultrasound spatial compound imaging method as described above when the computer program instructions are run.

[0017] The present invention introduces automatic weight allocation into ultrasonic spatial composite imaging, and assigns weights to the feature maps corresponding to the target multi-channel image data. This weight is non-fixed, which helps to automatically improve the signal-to-noise ratio according to the situation of the image data and retain important detail features, thereby helping to solve the problem of weakening or even disappearance of diagnostically significant signs such as lateral acoustic shadows due to fixed weights.

[0018] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features, and advantages of the present invention will become more apparent through a more detailed description of the embodiments of the present invention with reference to the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and are not intended to limit the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 is a schematic flow chart of an ultrasonic spatial compound imaging method according to an embodiment of the present invention;

[0021] Figure 2 Schematic diagram of the principle of ultrasonic spatial compound imaging according to an embodiment of the present invention;

[0022] Figure 3 is a schematic block diagram of an ultrasonic spatial compound imaging device according to an embodiment of the present invention;

[0023] Figure 4 is a schematic block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present invention more apparent, exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0025] As mentioned above, existing technologies primarily use preset fixed weights for image synthesis, which can weaken or even eliminate diagnostically significant features such as lateral acoustic shadowing. Embodiments of the present invention provide an ultrasound spatial composite imaging solution that automatically assigns weights to ultrasound image data through an image processing module and fuses feature maps based on the assigned weights.

[0026] According to one aspect of the present invention, a method for ultrasonic spatial compound imaging is provided.

[0027] See also Figure 1 As shown in FIG, it is a schematic flow chart of an ultrasonic spatial compound imaging method according to an embodiment of the present invention. Figure 1 As shown, the ultrasound spatial compound imaging method includes steps S1, S2 and S3.

[0028] In step S1, at least two groups of target ultrasonic image data are acquired. The at least two groups of target ultrasonic image data are acquired by performing echo signal processing on at least two groups of target ultrasonic echo signals. The at least two groups of target ultrasonic echo signals correspond to different acoustic beam incident directions.

[0029] Specifically, in step S1, the ultrasonic probe in the ultrasonic imaging device can emit acoustic beams at different angles toward the same target object. The target ultrasonic echo signals reflected by the target object are transmitted to the ultrasonic probe, where they are processed by the ultrasonic imaging device to form target ultrasonic image data. The ultrasonic probe can be a convex array probe, a circular array probe, or the like. Exemplarily, the echo signal processing process can include: denoising the target ultrasonic echo signal; enhancing the intensity of the target ultrasonic echo signal through techniques such as filtering and logarithmic amplification; modulating and demodulating the intensity-enhanced target ultrasonic echo signal; and arranging the grayscale values ​​corresponding to each signal point in the demodulated signal (corresponding to the signal intensity value of the target ultrasonic echo signal) into lines to form target ultrasonic image data. The target ultrasonic image data can be non-visual two-dimensional image data, and the target ultrasonic image data can indicate features corresponding to the target object. The number of target ultrasonic image data groups can be any number greater than one, but preferably, the number of target ultrasonic image data groups is an odd number. Exemplarily, the number of target ultrasound image data groups can be less than or equal to a preset number, which is the maximum threshold for the number of target ultrasound image data groups participating in ultrasound spatial compound imaging. For example, if the preset number is 11, the number of target ultrasound image data groups can be 3, 5, 7, 9, or 11. Exemplarily, the at least two groups of ultrasound echo signals corresponding to the at least two groups of target ultrasound image data can be target ultrasound echo signals that satisfy an angular symmetry condition. The angular symmetry condition can include: within each group of target ultrasound echo signals, there exists a group of target ultrasound echo signals corresponding to an acoustic beam incident direction A, and any target ultrasound echo signal other than the target ultrasound echo signal corresponds to an acoustic beam incident direction B, and another target ultrasound echo signal corresponds to an acoustic beam incident direction C, satisfying angular symmetry with the acoustic beam incident direction A as the axis of symmetry. Those skilled in the art will appreciate the form and acquisition method of the target ultrasound echo signals used for ultrasound spatial compound imaging, and will not be elaborated herein.

[0030] By way of example and not limitation, before step S2, and during or after acquiring at least two sets of target ultrasound image data, the target ultrasound image data may also be angle-corrected. When an ultrasound probe is used to transmit ultrasound waves, because the transmission direction of the ultrasound waves is not perpendicular to the surface of the target object, the target ultrasound image data corresponding to the reflected target ultrasound echo signal is stored with an incident direction perpendicular to the surface of the target object. Therefore, the angle data of the target object recorded in the unangle-corrected target ultrasound image data is deflected relative to the actual position of the target object. In a specific embodiment, phase correction can be performed on the target ultrasound image data based on the propagation time difference of the target ultrasound echo signal. The phase correction method can be, for example, a theoretical method based on approximate ray acoustics or a numerical calculation method based on time reversal.

[0031] In step S2, channel stitching is performed on at least two groups of target ultrasound image data to obtain target multi-channel image data.

[0032] In step S2, the channels of each set of target ultrasound image data can be spliced ​​together through a channel stitching operation to obtain a set of multi-channel image data, referred to as target multi-channel image data. In one embodiment of the present invention, the ultrasound probe may include at least one array element. The ultrasound probe uses at least some of the array elements to transmit ultrasound waves along different acoustic beam incidence directions at different times, and uses at least some of the array elements to receive a set of target ultrasound echo signals at each time. Each set of target ultrasound echo signals may correspond to only one channel, which contains the signal intensity value of the target ultrasound echo signal. Accordingly, each set of target ultrasound image data may correspond to only one channel, which contains the grayscale value of the target ultrasound image data. For any set of target ultrasound image data, the grayscale value corresponding to the set of target ultrasound image data corresponds to the signal intensity value of the target ultrasound echo signal corresponding to the set of target ultrasound image data. The number of channels of the target multi-channel image data obtained by channel stitching the sets of target ultrasound image data is equal to the number of target ultrasound echo signal sets and the number of target ultrasound image data sets. Channel stitching can be implemented, for example, using a stitching function, and the embodiments of the present invention do not limit the specific stitching method. For example, the target ultrasonic image data corresponding to the target ultrasonic echo signal may be arranged according to the incident angle of the target ultrasonic echo signal, thereby achieving channel stitching.

[0033] In step S3, the target multi-channel image data is input into an image processing module to obtain a target spatial composite image; the image processing module includes a spatial composite network, in which the feature maps corresponding to the target multi-channel image data are weighted in at least one dimension, and the feature maps corresponding to the target multi-channel image data are fused based on the assigned weights to obtain fused image information, and the fused image information is used to obtain the target spatial composite image.

[0034] In step S3, the spatial composite network may be a trained convolutional neural network. In one embodiment, the spatial composite network may include a convolutional block attention module (CBAM) structure. Weight allocation may include: determining the weight of the feature map corresponding to the target multi-channel image data in at least one dimension. Exemplarily, the at least one dimension may include a channel dimension and / or a spatial dimension, that is, the feature map corresponding to the target multi-channel image data may be weighted in the channel dimension and / or weighted in the spatial dimension. After the weight allocation, the feature maps corresponding to the target multi-channel image data may be fused according to the allocated weights to obtain fused image information.

[0035] According to the ultrasonic spatial compound imaging method of an embodiment of the present invention, automatic weight allocation is introduced into ultrasonic spatial compound imaging, and weights are allocated to the feature maps corresponding to the target multi-channel image data. This weight is non-fixed, which helps to automatically improve the signal-to-noise ratio according to the situation of the image data and retain important detail features, thereby helping to solve the problem of weakening or even disappearance of diagnostically significant signs such as side acoustic shadows due to fixed weights.

[0036] Exemplarily, the at least one dimension comprises a channel dimension and / or a spatial dimension.

[0037] Exemplarily, an embodiment of the present invention can learn the feature weights of target multi-channel image data based on the attention mechanism, wherein the attention mechanism learning can be channel attention learning for the target multi-channel image data alone, spatial attention learning for the target multi-channel image data alone, or a combined learning of channel attention and spatial attention for the target multi-channel image data.

[0038] By allocating weights in the channel dimension, it is possible to adaptively allocate weights to image data corresponding to different sound beam incident directions, which helps to improve the signal-to-noise ratio based on the global content corresponding to all image data and retain more important obscured detail features. By allocating weights in the spatial dimension, it is possible to adaptively allocate weights to position features corresponding to different image data, which helps to retain important features within the image data. By combining weight allocation in the channel dimension with weight allocation in the spatial dimension, it is possible to take into account the overall weights of image data in different directions and the allocation of position weights to different regions in the same direction. This not only achieves the effects of the above two weight allocations, but also helps to reduce the synthesis traces of the resulting spatial composite image.

[0039] Exemplarily, in step S3, the target multi-channel image data is input into the image processing module to obtain a target spatial composite image, including:

[0040] The spatial composite network performs the following operations:

[0041] Perform feature extraction on the target multi-channel image data to obtain an original feature map of the target multi-channel image data;

[0042] Assign channel attention weights and / or spatial attention weights to the original feature map;

[0043] Performing a channel attention operation and / or a spatial attention operation on the original feature map based on the assigned channel attention weight and / or spatial attention weight to obtain a calculated feature map;

[0044] generating fused image information based on the calculated feature map;

[0045] The fused image information is a target space composite image, or the target multi-channel image data is input into the image processing module to obtain the target space composite image, which also includes: generating the target space composite image based on the fused image information.

[0046] See also Figure 2 , which is a schematic diagram of the principle of ultrasound spatial compound imaging according to one embodiment of the present invention. By way of example and not limitation, the spatial compound network in the embodiment of the present invention can be implemented using a neural network with an encoder-decoder architecture. Figure 2 Where n is the number of spatial compound angles, equivalent to the number of target ultrasonic echo signals, and also the number of target ultrasonic image data groups participating in channel stitching. K represents the sequence number of the target ultrasonic image data. As described above, the target ultrasonic image data corresponding to each spatial compound angle can be a group of single-channel image data. After channel stitching, each group of target ultrasonic image data forms target multi-channel image data. The image processing module can include a feature extraction submodule. When the target multi-channel image data is input into the image processing module, the feature extraction submodule can perform feature extraction on the target multi-channel image data to generate an original feature map corresponding to the target multi-channel image data.

[0047] In one example, for the original feature map, weight distribution can be performed on the channel dimension to determine its corresponding channel attention weight, and the channel attention operation can be performed on the original feature map using the assigned channel attention weight to obtain a calculated feature map. In another example, for the original feature map, weight distribution can be performed on the spatial dimension to determine its corresponding spatial attention weight, and the spatial attention operation can be performed on the original feature map using the assigned spatial attention weight to obtain a calculated feature map. In yet another example, for the original feature map, weight distribution can be performed on the channel dimension and the spatial dimension to determine its corresponding channel attention weight and spatial attention weight, and the channel attention operation and spatial attention operation can be performed on the original feature map using the assigned channel attention weight and spatial attention weight to obtain a calculated feature map. Fused image information can be generated based on the calculated feature map. For example, the fused image information can be obtained by decoding the calculated feature map.

[0048] In one example, the fused image information output by the spatial compounding network can be the target spatial compound image itself or the target spatial compound image data. When the fused image information output by the spatial compounding network is the target spatial compound image data, the target spatial compound image data can be further processed to obtain the corresponding target spatial compound image. In one embodiment, a digital scan converter (DSC) module can be integrated into the spatial compounding network. Those skilled in the art will understand that the DSC module is mainly used for coordinate conversion and data interpolation. In the spatial compounding network, after obtaining the calculated feature map, it can be decoded to obtain a decoding result, which can be regarded as non-visible target spatial compound image data. The DSC module in the spatial compounding network can further optimize the geometric shape and resolution of the target spatial compound image data, so that the output of the spatial compounding network is a visible target spatial compound image (the target spatial compound image is the image fusion information in this embodiment). In another embodiment, the image processing module can include a spatial compounding network and a DSC module, and the DSC module can be connected in series with the spatial compounding network. In this case, the spatial composite network can output fused image information, which is non-visual target spatial composite image data. The DSC module can further optimize the geometric shape and resolution of the target spatial composite image data output by the spatial composite network to obtain a visible target spatial composite image.

[0049] In the above scheme, learning the channel attention weights of the features corresponding to the target multi-channel image data can enable the spatial composite network to pay more attention to the feature channels with important information. By globally learning the feature maps corresponding to the target multi-channel image data, the risk of overfitting can be reduced; learning the spatial attention weights of the features corresponding to the target multi-channel image data can enable the spatial composite network to pay more attention to the local areas in the image data, better capture the detailed information of the image data, and improve the adaptability of the spatial composite network to complex scene ultrasound images.

[0050] Exemplarily, assigning channel attention weights to the original feature map includes:

[0051] Perform a first pooling operation along the spatial dimension on the original feature map to obtain a first feature map;

[0052] Input the first feature map into the multilayer perceptron to obtain a second feature map output by the multilayer perceptron;

[0053] Normalizing the eigenvalues ​​in the second feature map to obtain a normalized third feature map, wherein the eigenvalues ​​of the third feature map under each channel are channel attention weights corresponding one-to-one to each channel of the original feature map;

[0054] Perform channel attention operation on the original feature map based on the assigned channel attention weight to obtain an operated feature map, including:

[0055] Multiply the channel attention weight corresponding to each channel in the original feature map by the corresponding channel of the original feature map to obtain the calculated feature map.

[0056] The first pooling operation can be any pooling operation along the spatial dimension (width and height). Exemplarily, the first pooling operation can include performing global average pooling and / or global maximum pooling along the spatial dimension of the original feature map to obtain the first feature map. Figure 2 As shown, in a specific embodiment, the first pooling operation may include a global average pooling operation and a global maximum pooling operation. The number of channels in the first feature map may be equal to the number of channels in the original feature map. When the first feature map is input into a multi-layer perceptron (MLP), the MLP learns the common features between the channels corresponding to each first feature map, calculates the attention weights between the channels corresponding to each first feature map to measure the degree of correlation between the channels corresponding to each first feature map, and maps the first feature map into channel weight vectors of different sizes to form a second feature map. The number of channels in the second feature map is equal to that in the original feature map. The eigenvalues ​​of the second feature map are normalized to generate a third feature map. The number of channels in the third feature map is equal to that in the original feature map, and if any channel c1 is selected in the third feature map, there is a channel c2 in the original feature map corresponding to channel c1. The channel attention weight corresponding to each channel in the original feature map is multiplied by the corresponding channel in the original feature map to obtain a computed feature map. The computed feature map is decoded and output through the output head to obtain the final fused image information.

[0057] In an embodiment of the present invention, normalization can be implemented using any normalization function, such as a Softmax function or a Sigmoid function. More specifically, in an embodiment of the present invention, the number of channels of the first feature map is equal to the number of channels of the original feature map, but the width and height of the first feature map are compressed to 1, that is, if the original feature map is of size H*W*C, the first feature map obtained after the first pooling operation is of size 1*1*C, where C represents the number of channels, H represents the height, and W represents the width. The first pooling operation can be to perform only global average pooling or only global maximum pooling on each channel of the original feature map, or to perform global average pooling and global maximum pooling on each channel of the original feature map, respectively, to obtain a first pooled feature map obtained by global average pooling and a second pooled feature map obtained by global maximum pooling. The first feature map may include a first pooled feature map and / or a second pooled feature map. The multilayer perceptron may process the first pooled feature map obtained by global average pooling and / or the second pooled feature map obtained by global maximum pooling, and output a third pooled feature map corresponding to the first pooled feature map and / or a fourth pooled feature map corresponding to the second pooled feature map. The second feature map may include the third pooled feature map and / or the fourth pooled feature map.

[0058] Through the above scheme, the first pooling operation can extract key channel information and map the attention weight distribution from all channels to a few key channels containing important information. By learning the importance of each channel, the influence of channels with smaller weights can be further reduced to prevent overfitting.

[0059] Exemplarily, assigning spatial attention weights to the original feature map includes:

[0060] Perform a second pooling operation along the channel dimension on the original feature map to obtain a fourth feature map;

[0061] performing convolution on the fourth feature map, and normalizing eigenvalues ​​in the convolved feature map to obtain a normalized fifth feature map, wherein the eigenvalue corresponding to each of the plurality of first feature map regions in the fifth feature map represents a spatial attention weight of the corresponding second feature map region in the original feature map, and a position of the first feature map region in the fifth feature map is consistent with a position of the corresponding second feature map region in the original feature map;

[0062] Performing a spatial attention operation on the original feature map based on the assigned spatial attention weights to obtain an operated feature map, including:

[0063] Multiply the spatial attention weight corresponding to each second feature map area in the original feature map by the corresponding second feature map area of ​​the original feature map to obtain the calculated feature map.

[0064] The second pooling operation can be any pooling operation along the channel dimension. Figure 2 As shown, in a specific embodiment, the second pooling operation may include global maximum pooling and / or global average pooling. Global maximum pooling and global average pooling may be performed along the channel dimension of the original feature map to obtain a fifth pooled feature map subjected to global maximum pooling and a sixth pooled feature map subjected to global average pooling, respectively. The fourth feature map may include the fifth pooled feature map and the sixth pooled feature map. The fifth pooled feature map and the sixth pooled feature map are concatenated by channel and then output through convolution to obtain the fifth feature map. The fifth feature map has the same height and width as the original feature map, and the number of channels of the fifth feature map is 1. That is, if the original feature map is H*W*C, the fifth and sixth pooled feature maps obtained after the second pooling operation are H*W*C1 and H*W*C2 respectively. The feature map after channel splicing is H*W*(C1+C2), and the fifth feature map obtained after convolution is H*W*1, where C represents the number of channels of the original feature map, H represents the height, and W represents the width. C1 is the number of channels of the fifth pooled feature map obtained by global average pooling, and C2 is the number of channels of the sixth pooled feature map obtained by global maximum pooling. More specifically, in spatial attention learning, global average pooling can average the feature values ​​on the original feature map along the channel dimension, thereby reducing the number of channels. Global maximum pooling retains the maximum value of each channel of the original feature map, so the number of channels may not change after global maximum pooling. Therefore, C1<C, C2≤C. The fourth feature map after channel concatenation is convolved, and the eigenvalues ​​in the convolved fourth feature map are normalized to obtain a fifth feature map. The normalization function can be a Softmax function, a Sigmoid function, etc. The fifth feature map contains several feature map regions, and each feature map region together constitutes the fifth feature map. Similarly, the original feature map contains several feature map regions that constitute the original feature map. The several feature map regions contained in the fifth feature map are defined as first feature map regions, and the several feature map regions contained in the original feature map are defined as second feature map regions. Then, if any first feature map region a1 is selected, there exists a second feature map region a2 corresponding to the first feature map region a1, and the position of the first feature map region a1 relative to the fifth feature map is consistent with the position of the second feature map region a2 relative to the original feature map. The obtained eigenvalue of the first feature map region a1 represents the spatial attention weight of the corresponding second feature map region a2. Multiply the spatial attention weight corresponding to each second feature map area in the original feature map by the corresponding second feature map area of ​​the original feature map to obtain the calculated feature map, decode the calculated feature map and output it through the output head to obtain the final fused image information.

[0065] By adopting the above scheme, the second pooling operation can focus on the local area of ​​the target multi-channel image data, reduce the calculation of the spatial processing network on a large amount of data, reduce the complexity of the calculation, and improve the expression ability of the fused image information by downsampling the spatial features.

[0066] Exemplarily, the allocation of channel attention weights and spatial attention weights includes:

[0067] Perform the first pooling operation along the spatial dimension on the original feature map to obtain the sixth feature map;

[0068] Normalizing the eigenvalues ​​in the sixth feature map to obtain a normalized seventh feature map, wherein the eigenvalues ​​of the seventh feature map under each channel are channel attention weights corresponding one-to-one to each channel of the original feature map;

[0069] Perform a second pooling operation along the channel dimension on the original feature map to obtain the eighth feature map;

[0070] performing convolution on the eighth feature map, and normalizing eigenvalues ​​in the convolved feature map to obtain a normalized ninth feature map, wherein an eigenvalue corresponding to each of the plurality of third feature map regions in the ninth feature map represents a spatial attention weight of a corresponding fourth feature map region in the original feature map, and a position of the third feature map region in the ninth feature map is consistent with a position of the corresponding fourth feature map region in the original feature map;

[0071] Based on the assigned channel attention weights and spatial attention weights, a channel attention operation and a spatial attention operation are performed on the original feature map to obtain an operated feature map, including:

[0072] Multiply the channel attention weight corresponding to each channel in the original feature map by the corresponding channel of the original feature map, and multiply the spatial attention weight corresponding to each fourth feature map area in the original feature map by the corresponding fourth feature map area of ​​the original feature map to obtain the calculated feature map.

[0073] Please continue reading Figure 2As shown, in a specific embodiment, when the channel attention weights and spatial attention weights are assigned to the original feature map, the method for obtaining the channel attention weights of each channel of the original feature map is consistent with the method for obtaining the channel attention weights of each channel of the original feature map in the aforementioned process of assigning the channel attention weights to the original feature map, and the method for obtaining the spatial attention weights of each feature map region of the original feature map is consistent with the method for obtaining the spatial attention weights of each feature map region of the original feature map in the aforementioned process of assigning the spatial attention weights to the original feature map, and they are not repeated here. It is worth noting that the channel attention operation and the spatial attention operation performed on the original feature map are compound operations, that is, the obtained operation feature map satisfies: the channel attention weight allocation of each channel has been completed, and the spatial attention weight allocation of each feature map region has been completed.

[0074] By adopting the above scheme, the channel attention weight and the spatial attention weight are combined, which enables the spatial composite network to pay attention to both spatial and channel information during the learning process, and enables the features of the target multi-channel image data to be adaptively adjusted in both the channel dimension and the spatial dimension, so that the final fused image information can better display the feature information.

[0075] Exemplarily, the spatial composite network is trained in the following way:

[0076] Acquiring at least two sets of sample ultrasound image data and corresponding annotated spatial composite images, wherein the at least two sets of sample ultrasound image data are obtained by performing echo signal processing on at least two sets of sample ultrasound echo signals, and the at least two sets of sample ultrasound echo signals each correspond to a different acoustic beam incident direction;

[0077] Perform channel stitching on at least two sets of sample ultrasound image data to obtain sample multi-channel image data, wherein the at least two sets of sample ultrasound image data each have a separate channel;

[0078] The sample multi-channel image data is input into the image processing module to obtain the predicted spatial composite image; the loss value of the spatial composite network in the image processing module is calculated based on the labeled spatial composite image and the predicted spatial composite image to train the spatial composite network.

[0079] The acquisition methods and distribution of acoustic beam incident directions of the sample ultrasonic image data and the target ultrasonic image data are similar. The implementation method of channel-joining at least two groups of sample ultrasonic images to obtain sample multi-channel image data is similar to the implementation method of channel-joining at least two groups of target ultrasonic image data to obtain target multi-channel image data. The implementation method of inputting the sample multi-channel image data into the image processing module to obtain the predicted spatial composite image is similar to the implementation method of inputting the target multi-channel image data into the image processing module to obtain the target spatial composite image, and will not be repeated here. The at least two groups of sample ultrasonic echo signals can meet a condition similar to the angular symmetry condition satisfied by the target ultrasonic echo signal, that is, there is a group of sample ultrasonic echo signals in the at least two groups of sample ultrasonic echo signals, whose corresponding acoustic beam incident direction is α, and there is an acoustic beam incident direction β corresponding to any group of sample ultrasonic echo signals other than the group of sample ultrasonic echo signals, and an acoustic beam incident direction γ corresponding to another group of sample ultrasonic echo signals, which satisfies the angular symmetry formed with the acoustic beam incident direction α as the symmetry axis. The number of groups of sample ultrasonic image data can be an odd number and greater than 1. For example, the sample ultrasound image data may be angle corrected before channel stitching. In a specific embodiment, the sample ultrasound image data may be phase corrected based on the propagation time difference of the sample ultrasound echo signal. The phase correction method may be, for example, a theoretical method based on approximate ray acoustics or a numerical calculation method based on time reversal. Please continue to refer to Figure 2 As shown, the supervised true value (groundtruth, abbreviated as GT) is the pixel value of the annotated spatial composite image, and the target object corresponding to the annotated spatial composite image is the same as the target object corresponding to the sample ultrasound image data. Optionally, the annotated spatial composite image can be synthesized by multi-directional ultrasound scanning videos. During the ultrasound scanning process, the ultrasound signals from different directions and angles are integrated and processed, and each angle corresponds to several frames of ultrasound images. The parameters are manually adjusted to enable it to meet the imaging effect used as the supervised true value. The pixel values ​​of the predicted spatial composite image output by the image processing module and the pixel values ​​of the annotated spatial composite image can be substituted into the loss function to obtain the corresponding loss value, and the spatial composite network can be trained based on the loss value. The loss value can evaluate the difference between the predicted spatial composite image and the annotated spatial composite image. By minimizing the loss value, the spatial composite network can make the output fused image information correspond to the information corresponding to the input image data as much as possible.

[0080] In an embodiment where the spatial compounding network directly outputs a target spatial compound image, the spatial compounding network may also directly output a predicted spatial compound image. In this case, the loss value of the spatial compounding network may be calculated based on the annotated spatial compound image and the predicted spatial compound image output by the spatial compounding network to train the spatial compounding network. In an embodiment where the image fusion information output by the spatial compounding network is not the target spatial compound image, the image processing module may include additional modules in addition to the spatial compounding network, such as the above-mentioned DSC module. In this case, the parameters of the additional modules may be fixed when training the spatial compounding network. In an embodiment where the image fusion information output by the spatial compounding network is not the target spatial compound image, the loss value of the spatial compounding network may be calculated based on the annotated spatial compound image and the predicted spatial compound image output by the image processing module to train the spatial compounding network.

[0081] By adopting the above scheme, the parameters of the spatial compound network are adjusted according to the comparison results between the annotated spatial compound image and the predicted spatial compound image, so that when the image processing module containing the spatial compound network enters the testing or inference stage, the imaging effect of the obtained spatial compound image is better.

[0082] Exemplarily, the loss value of the spatial compounding network is the pixel distance between the predicted spatial compounding image and the annotated spatial compounding image, where the pixel distance is calculated by the mean square error.

[0083] Specifically, the pixels of the labeled spatial composite image can be used as the true supervisory value to calculate the loss value of the spatial composite network. By way of example and not limitation, mean squared error loss can be used for training. This loss value is simple to calculate and improves network training efficiency.

[0084] Exemplarily, the input end of the image processing module is connected in series to the output end of the image preprocessing module, wherein the image preprocessing module is used to perform channel stitching on at least two groups of target ultrasound images to obtain target multi-channel image data. The image preprocessing module has a specific number of input ends. When the number of groups of target ultrasound image data input into the image preprocessing module is less than a specific number, the input data corresponding to the input end without target ultrasound image data input is set to 0.

[0085] In a specific embodiment, the number of input terminals of the image preprocessing module is fixed, for example, 9 groups. When the number of groups of input target ultrasound image data is less than the number of input terminals of the image preprocessing module, for example, the number of groups of input target ultrasound image data is 7 groups, the input data corresponding to the two input terminals without target ultrasound image data input is defined as 0.

[0086] With this solution, the image preprocessing module maintains a fixed structure. When the number of target ultrasound image data sets input is less than the number of input terminals in the image preprocessing module, to ensure the integrity of the input data structure, the data corresponding to the input terminals where no target ultrasound image data is input is set to 0. This ensures that channel splicing can adapt to different numbers of target ultrasound image data sets. This allows the data input to the image preprocessing module to be dynamically variable, thereby dynamically changing the number of target ultrasound image data sets participating in ultrasound spatial composite imaging.

[0087] Exemplarily, performing channel stitching on at least two sets of target ultrasound image data to obtain target multi-channel image data includes:

[0088] If the number of the at least two sets of target ultrasound image data is greater than or equal to 3, performing channel splicing on the at least two sets of target ultrasound image data by setting the i-th set of target ultrasound image data in the middle channel to obtain target multi-channel image data;

[0089] Wherein, i=floor(n / 2)+1, floor(n / 2) represents rounding n / 2 down, n is the number of at least two groups of target ultrasonic image data, n is an odd number, and the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the im-th group of target ultrasonic image data and the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the i+m-th group of target ultrasonic image data satisfy angular symmetry with respect to the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the i-th group of target ultrasonic image data, and m=1,2,...,floor(n / 2).

[0090] In one example, there are n groups of target ultrasonic image data to be input, and the channel corresponding to the i-th group of target ultrasonic image data is used as the middle channel in the channel stitching process, where i=floor(n / 2)+1, and floor represents rounding down. For example, when n=3, i=2, indicating that the channel corresponding to the second group of target ultrasonic image data is the middle channel. The acoustic beam incident direction of the target ultrasonic echo signal corresponding to the first group of target ultrasonic image data and the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the third group of target ultrasonic image data are angularly symmetric with respect to the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the second group of target ultrasonic image data. When n=5, i=3, indicating that the channel corresponding to the third group of target ultrasonic echo signals is the middle channel. The acoustic beam incident directions of the target ultrasonic echo signals corresponding to the first group of target ultrasonic image data and the acoustic beam incident directions of the target ultrasonic echo signals corresponding to the fifth group of target ultrasonic image data are angularly symmetric with respect to the acoustic beam incident directions of the target ultrasonic echo signals corresponding to the third group of target ultrasonic image data, and the acoustic beam incident directions of the target ultrasonic echo signals corresponding to the second group of target ultrasonic image data and the acoustic beam incident directions of the target ultrasonic echo signals corresponding to the fourth group of target ultrasonic image data are angularly symmetric with respect to the acoustic beam incident directions of the target ultrasonic echo signals corresponding to the third group of target ultrasonic image data. Angular symmetry can be achieved, for example, by satisfying that, for the acoustic beam incident direction f of the i-th group of target ultrasonic echo signals, the acoustic beam incident direction p of the im-th group of target ultrasonic echo signals and the acoustic beam incident direction q of the i+m-th group of target ultrasonic echo signals satisfy that the angle between the straight line on which the acoustic beam incident direction p lies and the straight line on which the acoustic beam incident direction f lies is equal to the angle between the straight line on which the acoustic beam incident direction q lies and the straight line on which the acoustic beam incident direction f lies.

[0091] The i-th set of target ultrasound image data corresponds to the target ultrasound echo signal obtained when the acoustic beam incident direction is closest to perpendicular to the target object. This means that the i-th set of target ultrasound image data represents the target object with the highest degree of restoration, minimizing loss of detail. Therefore, the above approach can better display the structural details of the target object.

[0092] According to another aspect of the present invention, an ultrasonic spatial composite imaging device is provided. Figure 3 , which is a schematic block diagram of an ultrasonic spatial compound imaging device according to an embodiment of the present invention. As shown in the figure, the device includes an acquisition module 310 , a stitching module 320 and an input module 330 .

[0093] an acquisition module 310 for acquiring at least two sets of target ultrasound image data, wherein the at least two sets of target ultrasound image data are acquired by performing echo signal processing on at least two sets of target ultrasound echo signals, wherein the at least two sets of target ultrasound echo signals correspond to different acoustic beam incident directions;

[0094] a stitching module 320 configured to stitch at least two sets of target ultrasound image data through channels to obtain target multi-channel image data, wherein the at least two sets of target ultrasound image data each have a separate channel;

[0095] An input module 330 is used to input the target multi-channel image data into the image processing module to obtain a target spatial composite image;

[0096] Among them, the image processing module includes a spatial composite network, in which the feature maps corresponding to the target multi-channel image data are weighted in at least one dimension, and the feature maps corresponding to the target multi-channel image data are fused based on the assigned weights to obtain fused image information, and the fused image information is used to obtain the target spatial composite image.

[0097] Exemplarily, the at least one dimension may include a channel dimension and / or a spatial dimension.

[0098] Exemplarily, the spatial composite network is used to perform the following operations:

[0099] Perform feature extraction on the target multi-channel image data to obtain an original feature map of the target multi-channel image data;

[0100] Assign channel attention weights and / or spatial attention weights to the original feature map;

[0101] Performing a channel attention operation and / or a spatial attention operation on the original feature map based on the assigned channel attention weight and / or spatial attention weight to obtain a calculated feature map;

[0102] Generate fused image information based on the calculated feature map.

[0103] Exemplarily, the spatial composite network is trained in the following way:

[0104] Acquiring at least two sets of sample ultrasound image data and corresponding annotated spatial composite images, wherein the at least two sets of sample ultrasound image data are obtained by performing echo signal processing on at least two sets of sample ultrasound echo signals, and the at least two sets of sample ultrasound echo signals each correspond to a different acoustic beam incident direction;

[0105] Perform channel stitching on at least two sets of sample ultrasound image data to obtain sample multi-channel image data, wherein the at least two sets of sample ultrasound image data each have a separate channel;

[0106] The sample multi-channel image data is input into the image processing module to obtain a predicted spatial composite image. A loss value of a spatial composite network in the image processing module is calculated based on the annotated spatial composite image and the predicted spatial composite image to train the spatial composite network. For example, the loss value of the spatial composite network is the pixel distance between the predicted spatial composite image and the annotated spatial composite image, where the pixel distance is calculated using mean squared error.

[0107] Exemplarily, the ultrasonic spatial compound imaging apparatus may further include an image preprocessing module, wherein an input end of the image processing module is connected in series to an output end of the image preprocessing module, wherein the image preprocessing module is configured to perform channel splicing on at least two sets of target ultrasonic image data to obtain target multi-channel image data. The image preprocessing module has a specific number of input ends, and when the number of sets of target ultrasonic image data input to the image preprocessing module is less than the specific number, the input data corresponding to the input ends without target ultrasonic image data is set to 0.

[0108] According to another aspect of the present invention, an electronic device is also provided. Figure 4 , which is a schematic block diagram of an electronic device according to an embodiment of the present invention. As shown in the figure, electronic device 400 includes a processor 410 and a memory 420. Memory 420 stores computer program instructions, which, when executed by processor 410, are used to execute the ultrasound spatial compound imaging method according to an embodiment of the present invention.

[0109] According to another aspect of the present invention, a computer-readable storage medium is also provided. The storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the above-mentioned ultrasound spatial compound imaging method is implemented. The storage medium may include, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0110] According to yet another aspect of the present invention, a computer program product is provided, comprising computer program instructions, which are used to execute the above-mentioned ultrasound spatial compound imaging method when run.

[0111] A person skilled in the art will readily understand the implementation structure, working principle, and beneficial effects of the electronic device, storage medium, and computer program product by reading the above ultrasonic spatial composite imaging method. For the sake of brevity, detailed description is omitted here.

[0112] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely illustrative and are not intended to limit the scope of the present invention. Various changes and modifications may be made therein by those skilled in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as claimed in the appended claims.

[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0114] In the several embodiments provided herein, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple units or components may be combined or integrated into another device, or some features may be omitted or not implemented.

[0115] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0116] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the description of exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this approach to the present invention should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More precisely, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with fewer features than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0117] It will be understood by those skilled in the art that, except where mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus disclosed herein may be combined in any combination. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature providing the same, equivalent, or similar purpose.

[0118] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

[0119] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules of the ultrasonic spatial compound imaging method according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0120] It should be noted that the above embodiments illustrate rather than limit the invention, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0121] The above is merely a description of specific embodiments of the present invention, and the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be covered by the scope of protection of the present invention. The scope of protection of the present invention shall be based on the scope of protection of the claims.

Claims

1. An ultrasonic spatial composite imaging method, characterized in that: include: Acquiring at least two sets of target ultrasonic image data, wherein the at least two sets of target ultrasonic image data are obtained by performing echo signal processing on at least two sets of target ultrasonic echo signals, wherein the at least two sets of target ultrasonic echo signals correspond to different acoustic beam incident directions; performing channel stitching on the at least two sets of target ultrasound image data to obtain target multi-channel image data, wherein the at least two sets of target ultrasound image data each have a separate channel; Inputting the target multi-channel image data into an image processing module to obtain a target spatial composite image; In which, the image processing module includes a spatial composite network, in which the feature maps corresponding to the target multi-channel image data are weighted in at least one dimension, and the feature maps corresponding to the target multi-channel image data are fused based on the assigned weights to obtain fused image information, and the fused image information is used to obtain the target spatial composite image.

2. The method according to claim 1, characterized in that The at least one dimension comprises a channel dimension and / or a spatial dimension.

3. The method according to claim 2, characterized in that Inputting the target multi-channel image data into an image processing module to obtain a target spatial composite image includes: The following operations are performed by the spatial composite network: Performing feature extraction on the target multi-channel image data to obtain an original feature map of the target multi-channel image data; Allocating channel attention weights and / or spatial attention weights to the original feature map; Performing a channel attention operation and / or a spatial attention operation on the original feature map based on the assigned channel attention weight and / or the spatial attention weight to obtain a calculated feature map; generating the fused image information based on the calculated feature map; The fused image information is the target space composite image, or the step of inputting the target multi-channel image data into an image processing module to obtain the target space composite image further includes: generating the target space composite image based on the fused image information.

4. The method according to claim 3, characterized in that Allocating channel attention weights to the original feature map includes: Performing a first pooling operation along the spatial dimension on the original feature map to obtain a first feature map; Inputting the first feature map into a multilayer perceptron to obtain a second feature map output by the multilayer perceptron; Normalizing the eigenvalues ​​in the second feature map to obtain a normalized third feature map, wherein the eigenvalue of each channel of the third feature map is a channel attention weight corresponding to each channel of the original feature map; Performing a channel attention operation on the original feature map based on the assigned channel attention weight to obtain a calculated feature map, including: The channel attention weight corresponding to each channel in the original feature map is multiplied by the corresponding channel of the original feature map to obtain the calculated feature map.

5. The method according to claim 3, characterized in that Allocating spatial attention weights to the original feature map includes: Performing a second pooling operation along the channel dimension on the original feature map to obtain a fourth feature map; performing convolution on the fourth feature map, and normalizing eigenvalues ​​in the feature map obtained by the convolution to obtain a normalized fifth feature map, wherein the eigenvalue corresponding to each of the plurality of first feature map regions in the fifth feature map represents a spatial attention weight of a corresponding second feature map region in the original feature map, and a position of the first feature map region in the fifth feature map is consistent with a position of the corresponding second feature map region in the original feature map; Performing a spatial attention operation on the original feature map based on the assigned spatial attention weight to obtain a calculated feature map, comprising: The spatial attention weight corresponding to each second feature map area in the original feature map is multiplied by the corresponding second feature map area of ​​the original feature map to obtain the calculated feature map.

6. The method according to claim 3, characterized in that Allocating channel attention weights and spatial attention weights to the original feature map includes: Performing a first pooling operation along the spatial dimension on the original feature map to obtain a sixth feature map; Normalizing the eigenvalues ​​in the sixth feature map to obtain a normalized seventh feature map, wherein the eigenvalues ​​of the seventh feature map under each channel are channel attention weights corresponding one-to-one to each channel of the original feature map; Performing a second pooling operation along the channel dimension on the original feature map to obtain an eighth feature map; performing convolution on the eighth feature map, and normalizing eigenvalues ​​in the feature map obtained by the convolution to obtain a normalized ninth feature map, wherein the eigenvalue corresponding to each of the plurality of third feature map regions in the ninth feature map represents a spatial attention weight of a corresponding fourth feature map region in the original feature map, and a position of the third feature map region in the ninth feature map is consistent with a position of the corresponding fourth feature map region in the original feature map; Performing a channel attention operation and a spatial attention operation on the original feature map based on the assigned channel attention weight and the spatial attention weight to obtain a calculated feature map, including: Multiply the channel attention weight corresponding to each channel in the original feature map by the corresponding channel of the original feature map, and multiply the spatial attention weight corresponding to each fourth feature map area in the original feature map by the corresponding fourth feature map area of ​​the original feature map to obtain the calculated feature map.

7. The method according to any one of claims 1 to 6, characterized in that The spatial composite network is trained in the following way: Acquiring at least two sets of sample ultrasound image data and corresponding annotated spatial composite images, wherein the at least two sets of sample ultrasound image data are obtained by performing echo signal processing on at least two sets of sample ultrasound echo signals, wherein the at least two sets of sample ultrasound echo signals correspond to different acoustic beam incident directions; Perform channel stitching on the at least two groups of sample ultrasound image data to obtain sample multi-channel image data, wherein the at least two groups of sample ultrasound image data each have a separate channel; Inputting the sample multi-channel image data into the image processing module to obtain a predicted spatial composite image; A loss value of the spatial compound network in the image processing module is calculated according to the labeled spatial compound image and the predicted spatial compound image to train the spatial compound network.

8. The method according to claim 7, characterized in that The loss value of the spatial compounding network is the pixel distance between the predicted spatial compounding image and the annotated spatial compounding image, wherein the pixel distance is calculated by the mean square error.

9. The method according to any one of claims 1 to 6, characterized in that The input end of the image processing module is connected in series to the output end of the image preprocessing module, wherein the image preprocessing module is used to perform channel splicing on the at least two groups of target ultrasound images to obtain the target multi-channel image data. The image preprocessing module has a specific number of input ends. When the number of groups of target ultrasound image data input to the image preprocessing module is less than the specific number, the input data corresponding to the input end without target ultrasound image data input is set to 0.

10. The method according to any one of claims 1 to 6, characterized in that The channel stitching of the at least two sets of target ultrasound image data to obtain target multi-channel image data includes: If the number of the at least two groups of target ultrasound image data is greater than or equal to 3, performing channel splicing on the at least two groups of target ultrasound image data by setting the i-th group of target ultrasound image data in the middle channel to obtain target multi-channel image data; Wherein, i=floor(n / 2)+1, floor(n / 2) represents rounding n / 2 down, n is the number of the at least two groups of target ultrasonic image data, n is an odd number, and the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the im-th group of target ultrasonic image data and the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the i+m-th group of target ultrasonic image data satisfy angular symmetry with respect to the acoustic beam incident direction of the target ultrasonic echo signal corresponding to the i-th group of target ultrasonic image data, and m=1,2,...,floor(n / 2).

11. An ultrasonic spatial composite imaging device, characterized in that: include: an acquisition module, configured to acquire at least two sets of target ultrasonic image data, the at least two sets of target ultrasonic image data being obtained by performing echo signal processing on at least two sets of target ultrasonic echo signals, the at least two sets of target ultrasonic echo signals each corresponding to a different acoustic beam incident direction; a splicing module, configured to perform channel splicing on the at least two sets of target ultrasound image data to obtain target multi-channel image data, wherein the at least two sets of target ultrasound image data each have a separate channel; An input module, configured to input the target multi-channel image data into an image processing module to obtain a target spatial composite image; In which, the image processing module includes a spatial composite network, in which the feature maps corresponding to the target multi-channel image data are weighted in at least one dimension, and the feature maps corresponding to the target multi-channel image data are fused based on the assigned weights to obtain fused image information, and the fused image information is used to obtain the target spatial composite image.

12. An electronic device comprising: A processor and a memory, wherein the memory stores computer program instructions, characterized in that the computer program instructions are used by the processor to execute the ultrasonic spatial compound imaging method according to any one of claims 1 to 10 when the processor is running.

13. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the ultrasound spatial compound imaging method according to any one of claims 1 to 10 when running.

14. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the ultrasound spatial compound imaging method according to any one of claims 1 to 10 when being run.

Images