Ultrasound image seamless stitching method, system and computer-readable storage medium

By using automated seamless stitching technology for ultrasound images, and employing methods such as convolutional basis tracking denoising and neural networks, the problem of small imaging area in ultrasound images has been solved, enabling comprehensive observation and accurate diagnosis of large organs, and improving image quality and diagnostic efficiency.

CN114066734BActive Publication Date: 2024-11-26ZHEJIANG QIUSHI INST OF MATHEMATICAL MEDICINE +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111394867.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-23
Publication Date
2024-11-26
Estimated Expiration
2041-11-23

AI Technical Summary

Technical Problem

Existing ultrasound imaging technology suffers from a narrow imaging area for large organs, making it difficult for clinicians to observe the entire organ and potentially leading to missed lesions. Furthermore, existing enlarged imaging methods result in decreased image quality and high costs.

Method used

By employing techniques such as convolutional basis tracking denoising algorithm, convolutional neural network, super-limit learning machine, Euclidean distance metric and RANSAC algorithm, automated seamless stitching of ultrasound images is achieved. Through feature extraction in spatial domain and transform domain, a hybrid feature space is constructed, and mapping and adaptive linear smoothing are performed using the homography matrix with optimal parameters to obtain high-quality stitching results.

Benefits of technology

It enables automated and seamless stitching of ultrasound images, expands the imaging area, improves clinicians' ability to observe large organs, avoids missing lesions, and supports accurate diagnosis and treatment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114066734B_ABST
    Figure CN114066734B_ABST
Patent Text Reader

Abstract

The present invention provides a method, system and computer-readable storage medium for seamless stitching of ultrasonic images. The method for seamless stitching of ultrasonic images includes the following steps: preprocessing the image to be stitched based on a convolutional basis pursuit denoising algorithm; extracting spatial domain feature information and transform domain feature information; constructing a hybrid feature space using the spatial domain feature information and the transform domain feature information, and screening the feature information in the hybrid feature space based on an ultra-limited learning machine; matching the main feature information once based on the Euclidean distance measure, and matching the main feature information after the primary matching twice based on the RANSAC algorithm to calculate a homography matrix with optimal parameters; mapping and calculating the floating image in the preprocessed image to be stitched, thereby achieving registration; stitching the registered floating image and the reference image based on an adaptive linear smoothing strategy to obtain an ultrasonic image stitching result. The above technical solution plays an important technical support role in the precise diagnosis and treatment based on ultrasonic images and the development of high-end ultrasonic imaging equipment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of medical image processing and medical artificial intelligence, and in particular to a method and system for seamlessly stitching ultrasonic images and a computer-readable storage medium. Background Art

[0002] Ultrasound imaging has the advantages of being non-invasive, non-radioactive, real-time, convenient, and low scanning cost. It is currently widely used in the screening, diagnosis, and treatment of diseases of the human abdominal organs and superficial organs. For example, for typical diseases such as breast cancer and liver cancer, ultrasound images are the imaging examination method that medical institutions at all levels in my country give priority to, and are of great significance to the in-depth promotion of the national tiered diagnosis and treatment system. In addition, ultrasound images are also commonly used image guidance tools for the treatment of cancer using minimally invasive or interventional methods. For example, when using microwave / radiofrequency ablation to treat breast cancer and liver cancer, ultrasound images are usually used to formulate surgical plans and implement intraoperative navigation.

[0003] However, ultrasound images have problems such as low spatial resolution, low contrast, limited penetration, small imaging area, and easy to be affected by the doctor's operating skills. The problems of low spatial resolution, low contrast, and limited penetration can be solved by image super-resolution, image enhancement, scanning sequence design, and frequency adjustment. There are many research works. The problem of being easily affected by the doctor's operating skills can be solved by standardized training of physicians' operating skills.

[0004] For small organs such as the prostate and thyroid gland, the narrow imaging area of ​​ultrasound images will not have a significant impact. However, for large organs such as the liver and breast, the narrow imaging area of ​​ultrasound images will make it difficult for clinicians to observe the overall picture of the entire organ, which will lead to the risk of missing lesions. Although the existing advanced ultrasound imaging equipment has the function of expanding imaging, it leads to a decline in image quality, and the expanded imaging area is still very limited and cannot meet clinical needs. In order to effectively expand the imaging area of ​​ultrasound imaging, the ultrasound probe can be modified and related hardware devices can be added. However, this solution leads to a substantial increase in the price of ultrasound equipment and a significant slowdown in the imaging speed, so it is difficult to promote and apply.

[0005] In the past decade, a number of researchers have tried to use image stitching methods to overcome the problem of small ultrasound imaging area. Due to the difficulties in extracting ultrasound image feature information, designing mapping models, and eliminating stitching seams, there are relatively few representative research works in the field of using image stitching methods to overcome the problem of small ultrasound imaging area. There are still problems such as feature mismatching, obvious stitching seams, and inability to fully automate.

[0006] Therefore, a novel ultrasound image seamless stitching method, system and computer-readable storage medium are needed to realize automatic seamless stitching of ultrasound images and obtain high-quality stitching results. Summary of the invention

[0007] In order to overcome the above-mentioned technical defects, the purpose of the present invention is to provide a method, system and computer-readable storage medium for seamless stitching of ultrasound images, which play an important technical support role in the precise diagnosis and treatment based on ultrasound images and the development of high-end ultrasound imaging equipment.

[0008] The present invention discloses a method for seamlessly splicing ultrasonic images, comprising the following steps:

[0009] Acquire ultrasound sequence images, select at least two images to be stitched from the ultrasound sequence images, and pre-process the stitched images based on a convolutional basis pursuit denoising algorithm;

[0010] Extracting spatial domain feature information and transform domain feature information from the ultrasound images to be stitched based on the spatial domain feature extraction strategy and the transform domain feature extraction strategy respectively;

[0011] A hybrid feature space is constructed using the spatial domain feature information and transform domain feature information of the ultrasound images to be spliced, and the feature information in the hybrid feature space is screened based on an extreme learning machine to retain the main feature information;

[0012] The main feature information is matched once based on the Euclidean distance measurement, and the main feature information after the first matching is matched twice based on the RANSAC algorithm to calculate the homography matrix with optimal parameters;

[0013] A mapping model is established based on a homography matrix with optimal parameters, and a mapping calculation is performed on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is registered with the reference image in the preprocessed image to be stitched;

[0014] The registered floating image and the reference image are stitched together based on an adaptive linear smoothing strategy to obtain the ultrasound image stitching result.

[0015] Preferably, a sequence of ultrasonic images is collected, and at least two images to be stitched are selected from the sequence of ultrasonic images. The steps of preprocessing the images to be stitched based on a convolutional basis pursuit denoising algorithm include:

[0016] Acquire ultrasonic sequence images, and select two images to be stitched from the ultrasonic sequence images, wherein the images to be stitched are scanning results of scanning at a uniform speed at the same section angle and in the same direction;

[0017] One of the images to be stitched is used as the reference image I1, and the other image to be stitched is used as the floating image I2. Based on the convolutional basis pursuit denoising algorithm, the reference image I1 and the floating image I2 are denoised using the following formula:

[0018]

[0019] Where s represents the reference image I1, {d m} is an M-dimensional dictionary filter, {x m} is a coefficient map and has the same size as the reference image I1, λ is a regularization parameter, * is defined as a convolution operation, and s and {x m Each column in} is an N-dimensional vector, where N is the total number of pixels in the reference image I1; when s represents the floating image I2, {x m} has the same size as the floating image I2, and N is the total number of pixels in the floating image I2;

[0020] Based on the alternating direction multiplication method, we can solve the above formula {x m}, introducing additional variables {y m} as a constraint, rewrite the above formula as follows:

[0021]

[0022] Split the above rewritten formula into iterative form, and we get:

[0023]

[0024]

[0025]

[0026] Among them, ρ is the step size parameter;

[0027] Solve the {x as the encoding coefficient in the convolutional basis pursuit denoising algorithm m}, a 3×3 size median filter is used to filter {x m} filter, and then filter the result {x′ m} is inversely encoded to obtain the denoised reference image Is1 and the denoised floating image Is2.

[0028] Preferably, the steps of extracting spatial domain feature information and transform domain feature information from the ultrasound images to be stitched based on the spatial domain feature extraction strategy and the transform domain feature extraction strategy respectively include:

[0029] The convolutional layer and pooling layer of the convolutional neural network model expressed based on the following formula extract the spatial domain feature information of the denoised reference image Is1 and the denoised floating image Is2 respectively:

[0030] Convolutional layer formula:

[0031] Where * represents the convolution operation, l is the number of convolution layers, i represents the i-th neuron node in the l-1th layer, j represents the j-th neuron node in the lth layer, and M j is the set of input feature maps, is an intermediate node, representing the output of the l-1th layer and the input of the lth layer, represents the convolution kernel, b j represents the bias, and f is the activation function;

[0032] Pooling layer formula:

[0033] Among them, β is a scalar constant, q is the number of pooling layers, Represents the mean or maximum value of the matrix block of size n×n in the previous layer;

[0034] Based on the following image transformation formula and image inverse transformation formula:

[0035]

[0036]

[0037] Extract the transform domain feature information of the denoised reference image Is1 and the denoised floating image Is2, where x represents the reference image Is1 and the floating image Is2, respectively, and a t is the transform domain coefficient of x, and a0 is the low-frequency coefficient by default. When t=1,2,…,N-1, a t is the high frequency coefficient, <> represents the inner product operation, is a set of basis vectors, yes The dual basis vector of .

[0038] Preferably, the steps of constructing a hybrid feature space using spatial domain feature information and transform domain feature information in the ultrasound images to be stitched, and screening the feature information in the hybrid feature space based on an extreme learning machine to retain the main feature information include:

[0039] The spatial domain feature information and the transform domain feature information of the denoised reference image Is1 are placed in the same representation domain to construct a mixed feature space of the denoised reference image Is1;

[0040] The spatial domain feature information and the transform domain feature information of the denoised floating image Is2 are placed in the same representation domain to construct a mixed feature space of the denoised floating image Is2;

[0041] Let X, Y and Represent the vectors of the input layer, actual output layer and expected output layer of the ELM respectively. The objective function of the ELM is expressed as follows:

[0042]

[0043] Among them, β is the weight vector connecting the single hidden layer nodes and the output layer nodes in the extreme learning machine, and H represents the output matrix of the single hidden layer of the extreme learning machine. The formula is as follows:

[0044]

[0045] Among them, x1,…,x N is the element in the input layer vector X, ω is the weight vector connecting the input layer nodes and the single hidden layer nodes in the ELM, b is the threshold vector, N is the dimension of the input vector, is the number of nodes in a single hidden layer, and g is the activation function;

[0046] Solving the objective function of the ELM is transformed into an analytical solution for calculating β, so that β = H + Y, where H + is the Moore-Penrose generalized inverse of H, and Y is solved to obtain the feature classification result, thereby screening out the main feature information.

[0047] Preferably, the steps of performing a primary match on the main feature information based on the Euclidean distance measure, and performing a secondary match on the main feature information after the primary match based on the RANSAC algorithm to calculate a homography matrix with optimal parameters include:

[0048] Define the main feature information of the denoised reference image Is1 as Cluster1. Recorded as the feature point in Cluster1,

[0049] Define the main feature information of the denoised floating image Is2 as Cluster2. Recorded as the feature point in Cluster2,

[0050] For any feature point in Cluster2 Traverse from Cluster1 to find the feature points with the nearest neighbor and next neighbor in Euclidean distance and

[0051] when When and Defined as a match, otherwise and is defined as a mismatch, where ξ is a preset threshold.

[0052] Preferably, the step of performing a primary match on the main feature information based on the Euclidean distance measure, and performing a secondary match on the main feature information after the primary match based on the RANSAC algorithm to calculate a homography matrix with optimal parameters also includes:

[0053] The set of feature points obtained from a match is taken as a matching set S, and the matching set S is randomly divided into h groups, so that each group contains d pairs of feature matching points;

[0054] Take the feature matching points in the t-th group as the internal point set S t , calculate the interior point set S t The homography matrix H t , t=1,2,…,h; check one by one whether the d pairs of feature matching points in the t-th group are all suitable for their corresponding homography matrix H t ;

[0055] When all the d pairs of feature matching points in the t-th group are suitable for their corresponding homography matrix H t When , the feature matching points in the t-th group are taken as the internal point set S i , the corresponding homography matrix H t As the initial homography matrix H i , where i = 1;

[0056] Calculate the feature matching points of the remaining groups one by one and H i When the projection error of any feature matching point is less than the threshold thr, the feature matching point is added to S i middle;

[0057] Record the i-th iteration time S i The number of interior points Q in i ;

[0058] Repeat the above process until the preset maximum number of iterations N is reached, where i = 1, 2, ..., N;

[0059] Select the largest Q i value, lock the homography matrix H used in the corresponding i-th iteration i , and the homography matrix H i Recorded as the homography matrix with optimal parameters.

[0060] Preferably, a mapping model is established based on a homography matrix with optimal parameters, and a mapping calculation is performed on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is aligned with the reference image in the preprocessed image to be stitched, comprising:

[0061] The homography matrix H is selected based on the following formula i :

[0062]

[0063] Establish a homography matrix H i The mapping model is expressed as follows:

[0064]

[0065] Where [xy] and [x′ y′] are the coordinates before and after mapping of the denoised floating image Is2, respectively, to obtain the mapped and denoised floating image

[0066] Preferably, the step of stitching the registered floating image and the reference image based on an adaptive linear smoothing strategy to obtain an ultrasonic image stitching result includes:

[0067] The denoised reference image Is1 and the mapped and denoised floating image Place them in the same coordinate system, search for the maximum and minimum values ​​of the horizontal and vertical axes respectively; take the maximum value of the horizontal axis as the ultrasound image stitching result Is result The right edge of the horizontal axis is taken as the minimum value of the horizontal axis as the ultrasound image stitching result Is result The maximum value of the vertical axis is taken as the ultrasound image stitching result Is result The minimum value of the vertical axis is taken as the ultrasound image stitching result Is result The upper edge of

[0068] For ultrasound image stitching result Is result For each pixel point in the image, the corresponding coordinates are calculated by the above mapping model, and the pixel value at each coordinate is calculated by bilinear interpolation method;

[0069] Assume that the ultrasound image stitching result Is result The horizontal length of the overlapping area in is Distance, and the adaptive smoothing factor α is set, where α∈[0,1]. For the pixel point P with a distance of dist from the leftmost end of the overlapping area, the value formula of α is as follows:

[0070]

[0071] Establish ultrasound image stitching result Isresult The pixel value p of the pixel point P in the overlapping area result The solution formula for (x,y) is as follows:

[0072]

[0073] Among them, p1(x, y) is the pixel value of the corresponding pixel P1 in the denoised reference image Is1, and p2(x, y) is the floating image of the pixel P after mapping and denoising. The pixel value of the corresponding pixel point P2 in the image is obtained to obtain the ultrasonic image stitching result Is result .

[0074] The present invention also discloses an ultrasonic image stitching system, comprising:

[0075] A preprocessing module collects ultrasound sequence images, selects at least two images to be stitched from the ultrasound sequence images, and preprocesses the stitched images based on a convolutional base pursuit denoising algorithm;

[0076] An extraction module extracts spatial domain feature information and transform domain feature information from the ultrasound image to be spliced ​​based on a spatial domain feature extraction strategy and a transform domain feature extraction strategy respectively;

[0077] A screening module constructs a mixed feature space using the spatial domain feature information and the transform domain feature information of the ultrasound image to be spliced, and screens the feature information in the mixed feature space based on an extreme learning machine to retain the main feature information;

[0078] The matching module performs a primary match on the main feature information based on the Euclidean distance measurement, and performs a secondary match on the main feature information after the primary match based on the RANSAC algorithm to calculate the homography matrix with optimal parameters;

[0079] A mapping module, which establishes a mapping model based on a homography matrix with optimal parameters, and performs mapping calculation on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is registered with the reference image in the preprocessed image to be stitched;

[0080] The stitching module stitches the registered floating image and the reference image based on an adaptive linear smoothing strategy to obtain the ultrasound image stitching result.

[0081] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for seamlessly stitching ultrasonic images as described above is implemented.

[0082] Compared with the prior art, the above technical solution has the following beneficial effects:

[0083] 1. Adopting theoretical technologies such as Convolutional Neural Network (CNN), Convolutional Basis Pursuit De-Noising (CBPDN) algorithm, Extreme Learning Machine (ELM), Euclidean distance measure, Random Sample Consensus (RANSAC) algorithm, image transformation method, adaptive linear smoothing strategy, etc., it can realize automatic seamless stitching of ultrasound images and obtain high-quality stitching results, which effectively makes up for the disadvantage of narrow imaging area of ​​ultrasound imaging;

[0084] 2. It helps clinical physicians to observe the organs and tissues of the human body more comprehensively and accurately, avoiding the omission of lesions. It is of great value for accurate diagnosis and treatment based on ultrasound images (especially for lesions in large areas such as the breast and liver) and the development of high-end ultrasound imaging equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A schematic diagram of a flow chart of a method for seamlessly stitching ultrasound images in accordance with a preferred embodiment of the present invention;

[0086] Figure 2 A schematic diagram of the splicing of overlapping areas in accordance with a preferred embodiment of the present invention;

[0087] Figure 3 A schematic diagram of a reference image, a floating image and a stitching result of an ultrasonic image to be stitched in accordance with a preferred embodiment of the present invention;

[0088] Figure 4 A schematic diagram of a reference image, a floating image and a stitching result of an X-ray image to be stitched in accordance with a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0089] The advantages of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments.

[0090] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0091] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0092] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0093] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0094] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0095] In the following description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module" and "component" can be used interchangeably.

[0096] See also Figure 1 , is a flow chart of a method for seamlessly stitching ultrasonic images in accordance with a preferred embodiment of the present invention. In this embodiment, the method for seamlessly stitching ultrasonic images includes the following steps:

[0097] S100: Acquire a sequence of ultrasound images, select at least two images to be stitched from the sequence of ultrasound images, and pre-process the images to be stitched based on a convolutional basis pursuit denoising algorithm;

[0098] Ultrasonic sequence images are also called dynamic videos, and the images to be spliced ​​can also be frames in the ultrasonic sequence images. Through the preprocessing method in step S100, the burrs and salt and pepper noise in the ultrasonic images to be spliced ​​can be filtered out without damaging the quality of the ultrasonic sequence images as much as possible. The Convolutional Basis Pursuit De-Noising (CBPDN) algorithm is a minimization optimization algorithm for solving unknown parameters with regularization terms under the L1 norm.

[0099] S200: extracting spatial domain feature information and transform domain feature information from the ultrasound images to be stitched based on the spatial domain feature extraction strategy and the transform domain feature extraction strategy respectively;

[0100] Regarding feature extraction in the spatial domain, the convolutional layer and pooling layer of the Convolutional Neural Network (CNN) are used to calculate the stitched ultrasound image to obtain spatial domain feature information.

[0101] Regarding feature extraction in the transform domain, a series of high-frequency sub-band coefficient matrices of the ultrasound image to be spliced ​​are obtained by using image transformation means, and then these high-frequency sub-band coefficient matrices are inversely transformed to obtain transform domain feature information.

[0102] S300: constructing a hybrid feature space using the spatial domain feature information and the transform domain feature information of the ultrasound image to be spliced, and filtering the feature information in the hybrid feature space based on an extreme learning machine to retain the main feature information;

[0103] The Extreme Learning Machine is a single-hidden layer feedforward neural network. Its characteristics are that the weights of the connection nodes between the input layer and the single hidden layer and the threshold of the single hidden layer can be randomly generated; the weights of the connection nodes between the single hidden layer and the output layer do not need to be adjusted iteratively multiple times, but can be solved once by means of singular value decomposition (SVD).

[0104] S400: performing a primary match on the main feature information based on the Euclidean distance measurement, and performing a secondary match on the main feature information after the primary match based on the RANSAC algorithm, so as to calculate a homography matrix with optimal parameters;

[0105] RANSAC is the abbreviation of RANdom SAmple Consensus. The basic assumption of the RANSAC algorithm is that the sample contains correct data (inliers, data that can be described by the model) and abnormal data (outlies, data that deviates far from the normal range and cannot fit the mathematical model), that is, the data set contains noise. These abnormal data may be caused by wrong measurements, wrong assumptions, wrong calculations, etc. At the same time, RANSAC also assumes that, given a set of correct data, there is a method to calculate the model parameters that conform to these data.

[0106] S500: establishing a mapping model based on a homography matrix with optimal parameters, and performing mapping calculation on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is aligned with the reference image in the preprocessed image to be stitched.

[0107] S600: stitching the registered floating image and the reference image based on an adaptive linear smoothing strategy to obtain an ultrasound image stitching result.

[0108] Adopting an adaptive linear smoothing strategy to stitch the registered floating image and the reference image can avoid discontinuous distortion of pixel values ​​in the overlapping area, thus realizing seamless stitching of ultrasound images fully automatically.

[0109] See also Figure 2-4 In a preferred embodiment, step S100 includes:

[0110] S110: Acquire a sequence of ultrasonic images, and select two images to be stitched from the sequence of ultrasonic images, wherein the images to be stitched are scanning results of scanning at a uniform speed at the same section angle and in the same direction;

[0111] The ultrasound images to be spliced ​​are preprocessed using the Convolutional Basis Pursuit De-Noising (CBPDN) algorithm to remove burrs and salt and pepper noise in the ultrasound images to avoid interference with the extraction of effective features. Considering that a group (two or more) of ultrasound images are spliced, they need to be spliced ​​in pairs, and in clinical practice, the demand for splicing two ultrasound images is the greatest. Therefore, this embodiment focuses on the situation of splicing two ultrasound images, but the present invention can be directly extended to the situation of splicing more than two ultrasound images, not limited to two. In this embodiment, the ultrasound physician is required to maintain the same cutting angle when scanning (that is, the angle between the ultrasound probe and the examination site remains unchanged), and scan at a uniform speed along the same direction.

[0112] S120: See Figure 3Taking breast ultrasound images as an example, one of the images to be stitched is used as the reference image I1, and the other image to be stitched is used as the floating image I2. Based on the convolutional basis pursuit denoising algorithm, the reference image I1 and the floating image I2 are denoised using the following formula:

[0113]

[0114] Where s represents the reference image I1, {d m} is a set of M-dimensional dictionary filters, {x m} is a coefficient map and has the same size as the reference image I1, λ is a regularization parameter, * is defined as a convolution operation, and s and {x m Each column in} is an N-dimensional vector, where N is the total number of pixels in the reference image I1. When s represents the floating image I2, {x m} has the same size as the floating image I2, N is the total number of pixels in the floating image I2, and the meanings of the other symbols are the same. It should be noted that in order to avoid ambiguity, the case where s represents the reference image I1 and the floating image I2 is described separately. Solving the dictionary filter {d m}, the K-means singular value decomposition (K-SVD) algorithm is used for dictionary learning.

[0115] S130: Solve {x in the above formula based on the Alternating Direction Method of Multipliers (ADMM) m}, introducing additional variables {y m} as a constraint, rewrite the above formula as follows:

[0116]

[0117] S140: Split the above rewritten formula into ADMM iterative form to obtain:

[0118]

[0119]

[0120]

[0121] Among them, ρ is the step size parameter, which is equivalent to the augmented Lagrangian parameter;

[0122] S150: Solve {x as the coding coefficient in the convolutional basis pursuit denoising algorithm m}, a 3×3 size median filter is used to filter {x m} filter, and then filter the result {x′ m} for inverse encoding (ie d m *x′ m ), we get the denoised reference image Is1 and the denoised floating image Is2.

[0123] Further, step S200 includes:

[0124] S210: The convolution layer and the pooling layer of the convolutional neural network model expressed based on the following formula respectively extract the spatial domain feature information of the denoised reference image Is1 and the denoised floating image Is2:

[0125] Convolutional layer formula:

[0126] Where * represents the convolution operation, l is the number of convolution layers, i represents the i-th neuron node in the l-1th layer, j represents the j-th neuron node in the lth layer, and M j is the set of input feature maps, is an intermediate node, representing the output of the l-1th layer and the input of the lth layer, represents the convolution kernel, b j represents the bias, f is the activation function, which is a nonlinear function;

[0127] Pooling layer formula:

[0128] Among them, β is a scalar constant, q is the number of pooling layers, It represents the mean or maximum value of the matrix block of the previous layer with a size of n×n, which can also be called average pooling or maximum pooling. By performing the convolution layer operation first and then the pooling layer operation, since the two are cascaded, the output result of the pooling layer is the spatial domain feature information about Is1 or Is2.

[0129] It should be noted that the convolution layer is intended to initially extract the spatial feature information of the image, and the pooling layer is intended to process and strengthen this feature information to improve the robustness of feature extraction. Regardless of which convolutional neural network model is used (for example: AlexNet, VGG-Net, ResNet, DenseNet, U-net, etc.), regardless of the number of convolutional layers and pooling layers of the network model, regardless of which activation function f is used (for example: ReLU function, Leaky ReLU function, Sigmoid function, tanh function, etc.), regardless of whether average pooling or maximum pooling is used, it is considered to be the spatial domain feature extraction method involved in the present invention.

[0130] S220: Based on the following image transformation formula and image inverse transformation formula:

[0131]

[0132]

[0133] Extract the transform domain feature information of the denoised reference image Is1 and the denoised floating image Is2, where x represents the reference image Is1 and the floating image Is2, respectively, and a t is the transform domain coefficient of x, and a0 is the low-frequency coefficient by default. When t=1,2,…,N-1, a t is the high frequency coefficient, < > represents the inner product operation, is a set of basis vectors, yes If x is written in the form of x(p,q) with coordinates, then a t Written as a t (m,n). Then a0(m,n) represents the low-frequency subband coefficient matrix of x, which is set to zero. t (m,n)t=1,2,…,N-1 represents the high frequency subband coefficient matrix, which is retained.

[0134] It should be noted that the image transformation means can be two-dimensional Fourier transform, two-dimensional wavelet transform, Curvelet transform, Ridgelet transform, Contourlet transform and Shearlet transform, etc. Regardless of the length of the basis vector and the number of high-frequency sub-band layers, they are all considered to be the transform domain feature extraction method involved in the present invention.

[0135] Further, step S300 includes:

[0136] S310: placing the spatial domain feature information and the transform domain feature information of the denoised reference image Is1 in the same representation domain to construct a mixed feature space of the denoised reference image Is1;

[0137] S320: placing the spatial domain feature information and the transform domain feature information of the denoised floating image Is2 in the same representation domain to construct a mixed feature space of the denoised floating image Is2;

[0138] S330: The ELM is essentially a single hidden layer feedforward neural network model. Represent the vectors of the input layer, actual output layer and expected output layer of the ELM respectively. The objective function of the ELM is expressed as follows:

[0139]

[0140] Among them, β is the weight vector connecting the single hidden layer nodes and the output layer nodes in the extreme learning machine, and H represents the output matrix of the single hidden layer of the extreme learning machine. The formula is as follows:

[0141]

[0142] Among them, x1,…,x N is the element in the input layer vector X, ω is the weight vector connecting the input layer nodes and the single hidden layer nodes in the ELM, b is the threshold vector, N is the dimension of the input vector, is the number of nodes in a single hidden layer, and g is the activation function;

[0143] S340: Since the optimal values ​​of ω and b do not need to be considered during the execution of the ELM, only random values ​​need to be assigned to them. Therefore, the objective function of solving the ELM can be transformed into an analytical solution for calculating β, so that β = H + Y, where H + is the Moore-Penrose generalized inverse of H, which can be solved by singular value decomposition (SVD). Once the analytical solution of β is obtained, Y is solved, and the feature classification result is obtained, thereby screening out the main feature information.

[0144] Further, step S400 includes:

[0145] S410: Define the main feature information of the denoised reference image Is1 as Cluster1. Recorded as the feature point in Cluster1,

[0146] S420: Define the main feature information of the floating image Is2 after denoising as Cluster2. Recorded as the feature point in Cluster2,

[0147] S430: For any feature point in Cluster2 Traverse from Cluster1 to find the feature points with the nearest neighbor and the next neighbor in Euclidean distance and

[0148] when When and Defined as a match, otherwise and is defined as a mismatch, where ξ is the preset threshold. The square root expression in the upper left corner of the formula refers to With the nearest neighbor feature point The Euclidean distance of the formula, the square root expression in the lower left corner refers to and the next nearest feature point The Euclidean distance of . r and c represent the row coordinate and column coordinate respectively. ξ can be preferably set to 0.5.

[0149] Considering that the RANSAC algorithm is a parameter estimation algorithm, the core idea is to randomly select a certain number of sample points from the observation data set containing inliers and outliers, estimate the homography matrix, and find the homography matrix with the largest number of inliers and the smallest estimation error through iterative updating, that is, to solve the optimal parameters of the homography matrix. Therefore, step S400 also includes:

[0150] S440: taking the feature point set obtained from the first matching as a first matching set S, and randomly dividing the first matching set S into h groups, so that each group contains d pairs of feature matching points;

[0151] S450: Take the feature matching points in the t-th group as the inner point set S t , calculate the interior point set S t The homography matrix H t , t=1,2,…,h; check one by one whether the d pairs of feature matching points in the t-th group are all suitable for their corresponding homography matrix H t ;

[0152] S460: When all d pairs of feature matching points in the t-th group are suitable for their corresponding homography matrix H t When , the feature matching points in the t-th group are taken as the internal point set S i , the corresponding homography matrix H t As the initial homography matrix H i , where i = 1;

[0153] S470: Calculate the feature matching points of the remaining groups one by one and H i When the projection error of any feature matching point is less than the threshold thr, the feature matching point is added to S i middle;

[0154] S480: Record the i-th iteration time S i The number of interior points Q in i ;

[0155] S490: Repeat the above process until a preset maximum number of iterations N is reached, where i = 1, 2, ..., N;

[0156] S4110: Select the largest Q i value, lock the homography matrix H used in the corresponding i-th iteration i , and the homography matrix H iRecorded as the homography matrix with optimal parameters.

[0157] Further, step S500 includes:

[0158] S510: Select the homography matrix H based on the following formula i :

[0159] It contains 8 parameters, and affine transformations including rotation, translation, scaling and projection can be represented and controlled by these 8 parameters. i It is equivalent to an affine transformation matrix.

[0160] S520: Establish an affine transformation matrix (i.e., homography matrix H i ) is represented as follows:

[0161]

[0162] Where [xy] and [x′ y′] are the coordinates before and after mapping of the denoised floating image Is2, respectively, to obtain the mapped and denoised floating image

[0163] Further, step S600 includes:

[0164] S610: In this step, first determine the ultrasound image stitching result Is result The denoised reference image Is1 and the mapped and denoised floating image Place them in the same coordinate system, search for the maximum and minimum values ​​of the horizontal and vertical axes respectively; take the maximum value of the horizontal axis as the ultrasound image stitching result Is result The right edge of the horizontal axis is taken as the minimum value of the horizontal axis as the ultrasound image stitching result Is result The maximum value of the vertical axis is taken as the ultrasound image stitching result Is result The minimum value of the vertical axis is taken as the ultrasound image stitching result Is result The upper edge of

[0165] It can be understood that registration refers to making the geometric information in two or more images match each other in space. Mapping refers to selecting one image as the reference image and the other image as the floating image in the registration process, using the reference image as a reference, and converting (mapping) the floating image in space, so as to achieve the matching of the geometric information of the two images in space.

[0166] S630: For the ultrasound image stitching result Is resultFor each pixel point in the image, the corresponding coordinates are calculated by the above mapping model, and the pixel value at each coordinate is calculated by bilinear interpolation method;

[0167] S640: See Figure 2 , using adaptive linear smoothing strategy to eliminate the seams in the overlapping area. Assuming that the ultrasound image stitching result Is result The horizontal length of the overlapping area in is Distance, and the adaptive smoothing factor α is set, where α∈[0,1]. For the pixel point P with a distance of dist from the leftmost end of the overlapping area, the value formula of α is as follows:

[0168]

[0169] S650: Establishing ultrasound image stitching result Is result The pixel value p of the pixel point P in the overlapping area result The solution formula for (x,y) is as follows:

[0170]

[0171] Among them, p1(x, y) is the pixel value of the corresponding pixel P1 in the denoised reference image Is1, and p2(x, y) is the floating image of the pixel P after mapping and denoising. The pixel value of the corresponding pixel point P2 in result (x, y) is calculated based on the value of α, so as to obtain the ultrasound image stitching result Is result At this point, the final seamless stitching result of the ultrasound image can be obtained.

[0172] Any of the above embodiments can be applied to the task of stitching X-ray images, see Figure 4 , showing an example of chest X-ray image stitching.

[0173] The present invention also discloses an ultrasonic image stitching system, comprising: a preprocessing module, which collects ultrasonic sequence images, selects at least two images to be stitched from the ultrasonic sequence images, and preprocesses the images to be stitched based on a convolutional basis pursuit denoising algorithm; an extraction module, which extracts spatial domain feature information and transform domain feature information from the ultrasonic images to be stitched based on a spatial domain feature extraction strategy and a transform domain feature extraction strategy, respectively; a screening module, which constructs a mixed feature space using the spatial domain feature information and the transform domain feature information of the ultrasonic images to be stitched, and screens the feature information in the mixed feature space based on an extreme learning machine to retain the main feature information ; A matching module performs a primary match on the main feature information based on the Euclidean distance measurement, and performs a secondary match on the main feature information after the primary match based on the RANSAC algorithm to calculate a homography matrix with optimal parameters; a mapping module establishes a mapping model based on the homography matrix with optimal parameters, and performs mapping calculation on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is aligned with the reference image in the preprocessed image to be stitched; a stitching module stitches the aligned floating image and the reference image based on an adaptive linear smoothing strategy to obtain an ultrasonic image stitching result.

[0174] The present invention further discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for seamlessly stitching ultrasonic images as described above is implemented.

[0175] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for seamlessly stitching ultrasonic images, characterized in that: The following steps are involved: Acquire ultrasonic sequence images, select at least two images to be spliced ​​from the ultrasonic sequence images, and pre-process the images to be spliced ​​based on a convolutional basis pursuit denoising algorithm; Extracting spatial domain feature information and transform domain feature information from the ultrasound images to be stitched based on the spatial domain feature extraction strategy and the transform domain feature extraction strategy respectively; A hybrid feature space is constructed using the spatial domain feature information and transform domain feature information of the ultrasound images to be spliced, and the feature information in the hybrid feature space is screened based on an extreme learning machine to retain the main feature information; Performing a primary match on the main feature information based on the Euclidean distance measurement, and performing a secondary match on the main feature information after the primary match based on the RANSAC algorithm, so as to calculate a homography matrix with optimal parameters; A mapping model is established based on a homography matrix with optimal parameters, and a mapping calculation is performed on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is registered with the reference image in the preprocessed image to be stitched; The registered floating image and the reference image are stitched together based on an adaptive linear smoothing strategy to obtain the ultrasound image stitching result.

2. The method for seamlessly stitching ultrasonic images according to claim 1, characterized in that: The steps of collecting ultrasound sequence images, selecting at least two images to be stitched from the ultrasound sequence images, and preprocessing the images to be stitched based on a convolutional basis pursuit denoising algorithm include: Acquire ultrasonic sequence images, and select two images to be stitched from the ultrasonic sequence images, wherein the images to be stitched are scanning results of scanning at a uniform speed at the same section angle and in the same direction; One of the images to be stitched is used as the reference image I1, and the other image to be stitched is used as the floating image I2. Based on the convolutional basis pursuit denoising algorithm, the reference image I1 and the floating image I2 are denoised using the following formula: Where s represents the reference image I1, {d m } is an M-dimensional dictionary filter, {x m } is a coefficient map and has the same size as the reference image I1, λ is a regularization parameter, * is defined as a convolution operation, and s and {x m Each column in} is an N-dimensional vector, where N is the total number of pixels in the reference image I1; when s represents the floating image I2, {x m } has the same size as the floating image I2, and N is the total number of pixels in the floating image I2; Based on the alternating direction multiplication method, we can solve the above formula {x m }, introducing additional variables {y m } as a constraint, rewrite the above formula as follows: Split the above rewritten formula into iterative form, and we get: Among them, ρ is the step size parameter; Solve the {x as the encoding coefficient in the convolutional basis pursuit denoising algorithm m }, a 3×3 size median filter is used to filter {x m } filter, and then filter the result {x′ m } is inversely encoded to obtain the denoised reference image Is1 and the denoised floating image Is2.

3. The method for seamlessly stitching ultrasonic images according to claim 2, characterized in that: The steps of extracting spatial domain feature information and transform domain feature information from the ultrasound images to be stitched based on the spatial domain feature extraction strategy and the transform domain feature extraction strategy respectively include: The convolutional layer and pooling layer of the convolutional neural network model expressed based on the following formula extract the spatial domain feature information of the denoised reference image Is1 and the denoised floating image Is2 respectively: Convolutional layer formula: Where * represents the convolution operation, l is the number of convolution layers, i represents the i-th neuron node in the l-1th layer, j represents the j-th neuron node in the lth layer, and M j is the set of input feature maps, is an intermediate node, representing the output of the l-1th layer and the input of the lth layer, represents the convolution kernel, b j represents the bias, and f is the activation function; Pond layer formula: Among them, β is a scalar constant, q is the number of pooling layers, Represents the mean or maximum value of the matrix block of size n×n in the previous layer; Based on the following image transformation formula and image inverse transformation formula: Extract the transform domain feature information of the denoised reference image Is1 and the denoised floating image Is2, where x represents the reference image Is1 and the floating image Is2, respectively, and a t is the transform domain coefficient of x, and a0 is the low-frequency coefficient by default. When t=1,2,…,N-1, a t is the high frequency coefficient, <> represents the inner product operation, is a set of basis vectors, yes The dual basis vector of .

4. The method for seamlessly stitching ultrasonic images according to claim 3, characterized in that: The steps of constructing a hybrid feature space by using the spatial domain feature information and the transform domain feature information in the ultrasound image to be stitched, and screening the feature information in the hybrid feature space based on an extreme learning machine to retain the main feature information include: The spatial domain feature information and the transform domain feature information of the denoised reference image Is1 are placed in the same representation domain to construct a mixed feature space of the denoised reference image Is1; The spatial domain feature information and the transform domain feature information of the denoised floating image Is2 are placed in the same representation domain to construct a mixed feature space of the denoised floating image Is2; Let X, Y and Represent the vectors of the input layer, actual output layer and expected output layer of the ELM respectively. The objective function of the ELM is expressed as follows: Among them, β is the weight vector connecting the single hidden layer nodes and the output layer nodes in the extreme learning machine, and H represents the output matrix of the single hidden layer of the extreme learning machine. The formula is as follows: Among them, x1,…,x N is the element in the input layer vector X, ω is the weight vector connecting the input layer nodes and the single hidden layer nodes in the ELM, b is the threshold vector, N is the dimension of the input vector, is the number of nodes in a single hidden layer, and g is the activation function; Solve the objective function of the ELM and convert it into an analytical solution for calculating β, so that β = H + Y, where H + is the Moore-Penrose generalized inverse of H, and Y is solved to obtain the feature classification result, thereby screening out the main feature information.

5. The ultrasonic image seamless splicing method according to claim 4, characterized in that: The steps of performing a primary match on the main feature information based on the Euclidean distance measurement and performing a secondary match on the main feature information after the primary match based on the RANSAC algorithm to calculate a homography matrix with optimal parameters include: Define the main feature information of the denoised reference image Is1 as Cluster1. Recorded as the feature point in Cluster1, Define the main feature information of the denoised floating image Is2 as Cluster2. Recorded as the feature point in Cluster2, For any feature point in Cluster2 Traverse from Cluster1 to find the feature points with the nearest neighbor and the next neighbor in Euclidean distance and when When and Defined as a match, otherwise and is defined as a mismatch, where ξ is a preset threshold.

6. The ultrasonic image seamless splicing method according to claim 5, characterized in that: The step of performing a primary match on the main feature information based on the Euclidean distance measure, and performing a secondary match on the main feature information after the primary match based on the RANSAC algorithm to calculate a homography matrix with optimal parameters also includes: The set of feature points obtained from a match is taken as a matching set S, and the matching set S is randomly divided into h groups, so that each group contains d pairs of feature matching points; Take the feature matching points in the t-th group as the internal point set S t , calculate the interior point set S t The homography matrix H t , t=1,2,…,h; check one by one whether the d pairs of feature matching points in the t-th group are all suitable for their corresponding homography matrix H t ; When all the d pairs of feature matching points in the t-th group are suitable for their corresponding homography matrix H t When , the feature matching points in the t-th group are taken as the internal point set S i , the corresponding homography matrix H t As the initial homography matrix H i , where i = 1; calculate the feature matching points in the remaining groups one by one and H i When the projection error of any feature matching point is less than the threshold thr, the feature matching point is added to S i middle; Record the i-th iteration time S i The number of interior points Q in i ; Repeat the above process until the preset maximum number of iterations N is reached, where i = 1, 2, ..., N; Select the largest Q i value, lock the homography matrix H used in the corresponding i-th iteration i , and the homography matrix H i Recorded as the homography matrix with optimal parameters.

7. The method for seamlessly stitching ultrasonic images according to claim 6, characterized in that: The steps of establishing a mapping model based on a homography matrix with optimal parameters, and performing mapping calculation on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is aligned with the reference image in the preprocessed image to be stitched include: The homography matrix H is selected based on the following formula i : Establish a homography matrix H i The mapping model is expressed as follows: Where [xy] and [x′y′] are the coordinates before and after mapping of the denoised floating image Is2, respectively, to obtain the mapped and denoised floating image 8. The method for seamlessly stitching ultrasonic images according to claim 7, characterized in that: The steps of stitching the registered floating image and the reference image based on the adaptive linear smoothing strategy to obtain the ultrasound image stitching result include: The denoised reference image Is1 and the mapped and denoised floating image Place them in the same coordinate system, search for the maximum and minimum values ​​of the horizontal and vertical axes respectively; take the maximum value of the horizontal axis as the ultrasound image stitching result Is result The right edge of the horizontal axis is taken as the minimum value of the horizontal axis as the ultrasound image stitching result Is result The maximum value of the vertical axis is taken as the ultrasound image stitching result Is result The minimum value of the vertical axis is taken as the ultrasound image stitching result Is result The upper edge of For ultrasound image stitching results Is result For each pixel point in the image, the corresponding coordinates are calculated by the above mapping model, and the pixel value at each coordinate is calculated by bilinear interpolation method; Assume that the ultrasound image stitching result Is result The horizontal length of the overlapping area in is Distance, and the adaptive smoothing factor α is set, where α∈[0,1]. For the pixel point P with a distance of dist from the leftmost end of the overlapping area, the value formula of α is as follows: Establish ultrasound image stitching result Is result The pixel value p of the pixel point P in the overlapping area result The solution formula for (x,y) is as follows: Among them, p1(x, y) is the pixel value of the corresponding pixel P1 in the denoised reference image Is1, and p2(x, y) is the floating image of the pixel P after mapping and denoising. The pixel value of the corresponding pixel point P2 in the image is obtained to obtain the ultrasonic image stitching result Is result .

9. An ultrasound image stitching system, characterized in that: include: A preprocessing module collects ultrasound sequence images, selects at least two images to be spliced ​​from the ultrasound sequence images, and preprocesses the images to be spliced ​​based on a convolutional basis pursuit denoising algorithm; An extraction module extracts spatial domain feature information and transform domain feature information from the ultrasound image to be spliced ​​based on a spatial domain feature extraction strategy and a transform domain feature extraction strategy respectively; A screening module constructs a mixed feature space using the spatial domain feature information and the transform domain feature information of the ultrasound image to be spliced, and screens the feature information in the mixed feature space based on an extreme learning machine to retain the main feature information; A matching module, performing a primary match on the main feature information based on a Euclidean distance measure, and performing a secondary match on the main feature information after the primary match based on a RANSAC algorithm, so as to calculate a homography matrix with optimal parameters; A mapping module, which establishes a mapping model based on a homography matrix with optimal parameters, and performs mapping calculation on the floating image in the preprocessed image to be stitched, so that the floating image in the preprocessed image to be stitched is registered with the reference image in the preprocessed image to be stitched; The stitching module stitches the registered floating image and the reference image based on an adaptive linear smoothing strategy to obtain the ultrasound image stitching result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for seamlessly stitching ultrasound images as described in any one of claims 1 to 8 is implemented.