Ultrasound image registration method, apparatus, and electronic device

Abstract

Embodiments of the present application disclose an ultrasound image registration method, device and electronic equipment, the method comprising the following steps: acquiring an abdominal ultrasound image of a patient as a floating image, and extracting a uterine fibroid contour in the floating image based on a trained network model; acquiring a corresponding abdominal gray-scale ultrasound as a reference image, and labeling an upper boundary of the uterine fibroid to obtain an incomplete contour; fitting the obtained complete and incomplete uterine fibroid contours into regular geometric contours; describing the regular geometric contours using a point set shape description method, and matching regular elliptical contours in the regular geometric contours to regular elliptical arc contours according to a shape difference matching rule; and describing the initial complete and incomplete uterine fibroid contours using the point set shape description method, matching the point sets of the initial complete and incomplete uterine fibroid contours according to the shape difference, and obtaining a transformation matrix to perform fine registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a uterine artery myoma branch ultrasound image registration method based on a uterine myoma contour, a device and an electronic equipment. BACKGROUND

[0002] Image registration is to find the spatial transformation between two images of the same object acquired under different conditions, so that the two images can be reasonably aligned through the transformation. Non-complete contour-based ultrasound image registration is a method of finding the same target in different images and aligning them using closed contours and non-closed or segmented contours as registration targets in medical ultrasound image registration. It is more commonly used in ultrasound-guided ablation surgery, especially in environments where ultrasound cannot be well imaged at a larger depth, such as gray-scale ultrasound-guided high-intensity focused ultrasound ablation. Intraoperative gray-scale ultrasound is used to detect blood flow in preoperative color Doppler flow ultrasound imaging, and to detect surgical conditions. Intraoperative ultrasound is limited by the structure of the ablation system, the poor quality of ultrasound imaging depth, and the difficulty of obtaining complete uterine myoma contours in the image. Therefore, using a non-complete contour ultrasound image registration method can enable accurate registration of gray-scale ultrasound images with large differences in contour information, enhancing the intraoperative guidance effect.

[0003] Non-complete contour-based image registration can be regarded as an extension of complete contour-based registration. The image registration method usually first extracts image features from the reference image and the floating image, then creates a cost function (such as mutual information, Euclidean distance, etc.) as a similarity measure of the image features, and finally uses an optimization algorithm to iteratively optimize the similarity measure to achieve the best similarity. Image feature extraction corresponds to different similarity measures. When focusing on the shape or contour of one or more uterine myomas in the image, the contour of the uterine myoma is usually extracted as a feature, and the difference between the uterine myoma contours of the reference image and the floating image is optimized to calculate the floating image transformation method. However, for gray-scale ultrasound images that cannot obtain complete uterine myoma contours, using a complete contour-based registration method to register gray-scale ultrasound images containing only partial uterine myoma contours is not good due to the sensitivity to the initial position. SUMMARY

[0004] To overcome the deficiencies of the prior art, the purpose of the present application is to provide an ultrasound image registration method, device and electronic equipment to achieve the purpose of gray-scale ultrasound image registration based on non-complete contours.

[0005] To achieve the above purpose, the present application provides an ultrasound image registration method, comprising the following steps:

[0006] Step S1, obtaining an abdominal ultrasound image of a patient as a floating image, and extracting a uterine fibroid contour in the floating image as an initial uterine fibroid complete contour based on a trained network model;

[0007] Step S2, obtaining a corresponding abdominal gray-scale ultrasound as a reference image, and labeling the upper boundary of the uterine fibroid to obtain an incomplete contour;

[0008] Step S3, fitting the obtained uterine fibroid complete contour and incomplete contour into a regular geometric contour;

[0009] Step S4, describing the regular geometric contour obtained in step S3 using a point set shape description method, and matching the regular elliptical contour in the regular geometric contour to the regular elliptical arc contour according to the shape difference matching rule;

[0010] Step S5, describing the initial uterine fibroid complete contour and the incomplete contour using a point set shape description method, and matching the point set of the initial uterine fibroid complete contour to the point set of the incomplete contour according to the shape difference matching rule to obtain a transformation matrix for precise registration.

[0011] Optionally, step S1 further comprises:

[0012] Step S100, obtaining an abdominal ultrasound image of a uterine fibroid patient with color Doppler blood flow imaging as a floating image;

[0013] Step S101, performing image preprocessing on the floating image;

[0014] Step S102, constructing a deep learning network model, dividing the preprocessed floating image into a training set, a validation set and a test set, and training the deep learning network model using the training set and the validation set;

[0015] Step S103, extracting the uterine fibroid contour in the floating image using the trained deep learning network model.

[0016] Optionally, in step S102, the constructed deep learning network model includes a generator and a discriminator, and the training or inference process thereof includes:

[0017] The encoder included in the generator of the deep learning network model encodes the network input to obtain a feature map;

[0018] The decoder included in the generator of the deep learning network model decodes the feature map to obtain a segmentation inference result as the generator output;

[0019] The preprocessed labeled uterine fibroid real segmentation and the generator output are jointly input into the discriminator, and the discriminator and the generator are trained simultaneously to make the generator and the discriminator against each other and then use the generator.

[0020] Optionally, step S2 further comprises:

[0021] Step S200, acquiring a gray-scale ultrasound image of the same patient as a reference image;

[0022] Step S201, pre-processing the reference image;

[0023] Step S202, manually labeling the uterine fibroid part contour of the intraoperative guidance gray-scale ultrasound image to obtain an incomplete contour point set.

[0024] Optionally, in step S3, the obtained uterine fibroid complete contour and incomplete contour are fitted into the regular elliptical contour and the regular elliptical arc contour using an elliptical contour fitting method.

[0025] Optionally, step S4 further comprises:

[0026] Step S400, using a shape context description method to describe the context shape of each point in the regular graph point set obtained in step S3, and two points with the smallest shape difference in the two point sets form a point pair;

[0027] Step S401, calculating a rigid transformation matrix between the two point sets;

[0028] Step S402, performing overall transformation on the floating image point set according to the calculated transformation matrix to complete pre-registration.

[0029] Optionally, step S400 further comprises:

[0030] Step S400a, taking one point q in the regular elliptical arc contour point set as the origin, establishing a polar coordinate system, and normalizing the positions of other points in the regular elliptical arc contour point set relative to point q to form a statistical histogram j j

[0031] Step S400b, performing the same operation as step S400a on the regular elliptical contour point set P to calculate and construct a statistical histogram sequence

[0032] Step S400c, using a bipartite graph matching method to find a matching method that can minimize the total cost with the sum of histogram cost functions as the standard, and performing point set matching into point pairs on the two point sets.

[0033] ​​Optionally, in step S5, the fine registration uses an iterative nearest algorithm, and on the basis of the pre-registration in step S4, the real contour segmented in step S1 is used to constantly update the transformation matrix with the average distance between the two sets of points as the optimization target, so that the registration state is reached when the iterative change is reduced to a certain threshold.

[0034] To achieve the above object, the application further provides an ultrasound image registration device, comprising:

[0035] A complete contour segmentation and extraction unit is configured to obtain an abdominal ultrasound image of a patient as a floating image, and extract a uterine fibroid contour in the floating image as an initial uterine fibroid complete contour based on a trained network model.

[0036] A non-complete contour acquisition unit is configured to obtain a corresponding abdominal gray-scale ultrasound as a reference image, and label an upper boundary of the uterine fibroid to obtain a non-complete contour.

[0037] A fitting unit is configured to fit the obtained uterine fibroid complete contour and non-complete contour into a regular geometric contour.

[0038] A pre-registration unit is configured to describe the regular geometric contour obtained by the fitting unit using a point set shape description method, and match a regular elliptical contour in the regular geometric contour to a regular elliptical arc contour according to a shape difference matching rule.

[0039] A fine registration unit is configured to describe the initial uterine fibroid complete contour and the non-complete contour using a point set shape description method, and match the initial uterine fibroid complete contour point set and the non-complete contour point set according to a shape difference matching rule and calculate a transformation matrix for fine registration.

[0040] To achieve the above object, the application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the ultrasound image registration method when executing the computer program.

[0041] Compared with the prior art, the ultrasound image registration method, device, storage medium and electronic device have the following beneficial effects:

[0042] (1) The application trains a deep learning-based ultrasound image segmentation network based on a generative adversarial network, and the method has strong fitting capability and can be applied to poor quality ultrasound images.

[0043] (2) The application directly extracts a registration uterine fibroid contour using a segmentation method, trains a network applicable to different modalities for segmentation of uterine fibroids, and then performs the next step of image registration based on the contour. The application is applicable to different imaging modalities.

[0044] (3) The present application can realize the matching between the complete contour and the non-complete contour, and further match the images with the same anatomical structure but different imaging effects;

[0045] (4) The present application can overcome the problem that the existing registration method is sensitive to the initial position, and can achieve good registration effect through the point set matching between the non-complete contour and the complete contour at any position as a starting point.

[0046] The technical solutions of the present application will be further described in detail below by means of the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application taken in conjunction with the accompanying drawings. The drawings provided in the specification and the embodiments of the present application together constitute a part of the detailed description of the present application and serve to explain the present application together with the detailed description of the present application and do not limit the present application. In the drawings, the same reference numerals refer to the same components or steps throughout the specification.

[0048] Figure 1 is a flowchart of an ultrasound image registration method provided by an exemplary embodiment of the present application;

[0049] Figure 2 is an architecture diagram of a deep learning network model constructed in an exemplary embodiment of the present application;

[0050] Figure 3 is an architecture diagram of a generator in an exemplary embodiment of the present application;

[0051] Figure 4 is a diagram showing that a polar coordinate system is established on a point in the contour point set Q in an exemplary embodiment of the present application;

[0052] Figure 5 is a diagram showing that the number of other points of the point set in different sector regions is counted to form a normalized statistical histogram in an exemplary embodiment of the present application.

[0053] Figure 6 is a structural block diagram of an ultrasound image registration device provided by an exemplary embodiment of the present application;

[0054] Figure 7 is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical solutions of the present application will be clearly and completely described below with reference to the accompanying drawings and the description of the present application. The described embodiments are part of the embodiments of the present application, but not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0056] It should be noted that the term "in the same plane" used in this document to describe the positional relationship between the reference image and the floating image does not mean that the reference image and the floating image are strictly in the same plane. The spatial planes of the two types of images can have a slight angle or positional offset. The medical images used in the examples also include, but are not limited to, medical images obtained from clinical ultrasound images, public data sets, etc.

[0057] In the description of the present application, "normalization", "ROI extraction", "gray scale change", "random left-right flipping", and "random noise" are generally referred to all operations that can achieve the target, and are operations in the data preprocessing in the following examples. For example, the "normalization" operation refers to transforming the data to the [0, 1] interval while preserving certain data characteristics, including but not limited to linear function normalization, 0-mean standardization, and inverse tangent function conversion.

[0058] In the description of the present application, the method of "shape context description" for describing the contour is a special case of the description method, and the contour description method includes but is not limited to the shape context description method.

[0059] Exemplary method

[0060] Figure 1 is a flowchart of an ultrasound image registration method provided by an exemplary embodiment of the present application. The present embodiment can be applied on an electronic device, such as a computer, a mobile phone, a tablet computer, a server, etc. Figure 1 As shown in the figure, the ultrasound image registration method of the present application comprises the following steps:

[0061] Step S1, obtaining an abdominal ultrasound image of a patient as a floating image, and extracting a uterine fibroid contour in the floating image as an initial complete uterine fibroid contour based on a trained network model.

[0062] Specifically, step S1 further comprises:

[0063] Step S100, obtaining an abdominal ultrasound image of a uterine fibroid patient with color Doppler flow imaging (CDFI) as a floating image.

[0064] Specifically, an abdominal ultrasound image of a patient with uterine fibroids is acquired as a floating image. The clinical ultrasound image with CDFI used in this example is collected using an ultrasound imaging device produced by General Electric Company, model Voluson 730 Expert.

[0065] Step S101, image preprocessing is performed on the floating image.

[0066] In this embodiment, the image preprocessing includes but is not limited to grayscale preprocessing, shape preprocessing, label preprocessing and image augmentation of the image. Specifically, the image preprocessing of the floating image includes but is not limited to outlining the uterine fibroid contour in the floating image, normalization, ROI cropping, random left-right inversion, random noise and random contrast enhancement.

[0067] Step S102, a deep learning network model is constructed, the preprocessed floating image is divided into a training set, a validation set and a test set, the deep learning network model is trained using the training set and the validation set, and the uterine fibroid contour in the floating image is extracted using the deep learning network model.

[0068] In this embodiment, after the image preprocessing of the floating image, the outlined data set is divided into a training set, a validation set and a test set according to a ratio of 6:2:2, so as to train the constructed deep learning network model.

[0069] As shown in Figure 2 , the constructed deep learning network model includes a generator and a discriminator, specifically, the ResRefineNet segmentation network is used as the generator to generate the inference result of image segmentation, and the PatchGAN is used as the discriminator to judge the true or false of the true value and the generated value, wherein the generator ResRefineNet is as shown in Figure 3 , the generator is divided into an encoder and a decoder, the encoder uses ResNet-18 as a basic encoder, and the decoder uses RefineNet as a basic decoder, the encoder output and the decoder input are both four layers, and the connection mode constitutes a segmentation network encoder-decoder structure, and the output of each layer of the encoder is input into a basic decoder. Figure 3

[0070] In this embodiment, the training or inference process of the deep learning network model is as follows:

[0071] Step 1: the encoder included in the generator in the deep learning network model encodes the network input to obtain a feature map;

[0072] Step 2: the decoder included in the generator in the deep learning network model decodes the feature map to obtain a segmentation inference result as the generator output; ​

[0073] Step 3: Input the pre-processed labeled real segment of uterine fibroids and the generator output into the discriminator, and train the discriminator and generator at the same time, so that the generator and discriminator compete against each other and thus achieve the use of the generator.

[0074] In other words, the pre-processed floating image training set is input into the deep learning network model, and its generator produces the predicted image segmentation result of uterine fibroids. Then, the generator's prediction is marked as false, and the real uterine fibroids are segmented as true. The results are input into the discriminator to train the discriminator, so that the generator and the discriminator can train against each other. The training is completed when the test set loss function reaches its minimum for the first time.

[0075] Step S103: Use the trained deep learning network model to extract the outline of uterine fibroids from the floating image.

[0076] Specifically, once the deep learning network model is trained, the unlabeled floating image is input into the trained deep learning network model to segment the uterine fibroids in the floating image and obtain the complete outline of the uterine fibroids. That is, the segmentation network in the deep learning network model is used to segment the floating image to obtain the complete outline of the uterine fibroids.

[0077] Step S2: Obtain the corresponding abdominal grayscale ultrasound image as a reference image and mark the upper boundary of the uterine fibroid.

[0078] Specifically, step S2 further includes:

[0079] Step S200: Obtain a grayscale ultrasound image of the same patient as a reference image.

[0080] In this embodiment, grayscale ultrasound images are used as surgical navigation during uterine artery fibroid branch ablation surgery using a grayscale ultrasound imaging device of model OPENO580 manufactured by Zhonghui Medical Technology Co., Ltd.

[0081] It should be noted that the ultrasound image with CDFI obtained in step S1 and the grayscale ultrasound image obtained in step S2 are in the same human body section.

[0082] Step S201: Preprocess the reference image.

[0083] Similarly, the image preprocessing includes, but is not limited to, grayscale preprocessing, shape preprocessing, label preprocessing, and image augmentation. Specifically, the preprocessing of the reference image includes, but is not limited to, gamma correction, contrast enhancement, and ROI cropping.

[0084] Step S202: Manually annotate the outline of the uterine fibroid on the intraoperative guided grayscale ultrasound image to obtain an incomplete outline point set.

[0085] In this embodiment, the upper half of the uterine fibroid is manually annotated with incomplete outlines, where incomplete outlines refer to non-closed curved outlines. This step is mainly done manually, and the upper boundary of the uterine fibroid is marked as the incomplete outline point set.

[0086] Step S3: Fit the obtained complete and incomplete contours of the uterine fibroids into a regular geometric contour.

[0087] The complete outline of the uterine fibroid refers to the outline of the uterine fibroid in the floating image extracted based on the trained network model in step S1. Specifically, in step S3, the obtained complete and incomplete outlines of the uterine fibroid are fitted into regular geometric outlines using regular graphics. The regular graphics fitting usually uses non-rotationally symmetric graphics such as rectangles, ellipses with equal short sides or unequal major and minor axes as the regular graphics fitting for the uterine fibroid.

[0088] In a preferred embodiment, the obtained complete and incomplete contours of the uterine fibroid are fitted into regular elliptical contours P and elliptical arc contours Q using an elliptical contour fitting method, wherein the elliptical contour P is a set of contour points in the floating image, and the elliptical arc contour Q is a set of incomplete contour points in the reference image.

[0089] Step S4: Use the point set shape description method to describe the regular geometric contour obtained in step S3, and match the regular elliptical contour P to the regular elliptical arc contour Q according to the shape difference matching.

[0090] In this embodiment, during the registration process, the elliptical arc contour Q remains stationary while the elliptical contour P moves, that is, the elliptical contour P is moved onto the elliptical arc contour Q according to the transformation matrix. Specifically, step S4 further includes:

[0091] Step S400: Use the shape context description method to perform context shape description on each point in the set of regular graphic points obtained in step S3, and form a point pair with the two points with the smallest shape difference between the two point sets.

[0092] Specifically, the point set shape description method used in this embodiment is the shape context description method. The step of using the shape context description method to perform context shape description on each point in the regular graphic point set obtained in step S3 further includes:

[0093] Step S400a: Using a point qj in the elliptical arc contour point set Q of the incomplete contour of the uterine fibroid as the origin, establish a polar coordinate system, count the positions of other points in the elliptical arc contour point set Q relative to point qj and normalize them to form a statistical histogram.

[0094] Figure 4This is a schematic diagram of establishing a polar coordinate system for a single point in a contour point set. That is, during the calculation process, all points in the contour need to be processed sequentially. Figure 4 Operations for establishing a coordinate system. For example... Figure 4 As shown, firstly, a polar coordinate system is established on a point in the point set Q of an elliptical arc contour with an incomplete contour. The tangent direction is taken as the zero polar axis, and the plane is divided into several sector regions according to the logarithmic interval of the polar moment and the equally spaced polar angles. Then, the number of other points in the same point set in different sector regions is counted (i.e., after establishing the polar coordinate system, the number of other points besides the origin of the polar coordinate system is counted in each sector region) to form a normalized statistical histogram. (That is, after establishing a coordinate system for each individual point, a histogram is generated by statistically analyzing the sector region, until the "establish coordinate system and perform statistical analysis" operation is completed for all points within the contour point set.) Figure 5 As shown, the horizontal axis represents the equally spaced polar angles, and the vertical axis represents the logarithmically spaced polar moments. The darker the color, the more midpoints the sector represented by that region. Finally, normalized statistical histograms are also constructed for other points in this set to form a statistical histogram sequence.

[0095] Step S400b: Perform the same operations as in step S400a on the regular elliptical outline point set P of the uterine fibroids to calculate and construct a statistical histogram sequence.

[0096] Step S400c: Using a bipartite graph matching method, based on the sum of the histogram cost functions, find the matching method that minimizes the total cost, and perform point set matching on two point sets to form point pairs.

[0097] In this embodiment, the matching between complete and incomplete contours uses the Hungarian algorithm. The mathematical description of the cost function of the Hungarian algorithm in this embodiment is as follows:

[0098]

[0099] Where c is the cost function, and K is the number of elements in the normalized statistical histogram, with each element representing a point in the histogram with respect to q. j Position information in a polar coordinate system established with the origin.

[0100] Step S401: Calculate the rigid transformation matrix between the two point sets.

[0101] Specifically, step S401 further includes:

[0102] Step S401a: After establishing the complete contour point set and the incomplete contour point set, the two point sets are centered, as shown below:

[0103]

[0104] Where P′ C With Q′ C The set of paired points is the decentralized set, and P′ and Q′ are the set of points paired after step S400c. and The mean of the coordinates of the paired point set.

[0105] Step S401b: Construct the decomposition matrix, whose mathematical description formula is:

[0106] S=Q′ C P′ C T

[0107] Where S is the decomposition matrix.

[0108] Step S401c: Perform singular value decomposition on the decomposition matrix. The mathematical description formula is as follows:

[0109] S=UΣV T

[0110] Where ∑ is a diagonal matrix and U is the matrix to be selected.

[0111] Step S401d: Calculate the rotation matrix and translation vector, whose corresponding mathematical description formula is as follows:

[0112]

[0113] Where R is the rotation matrix, t m It is a translation vector.

[0114] Step S401e: Construct the transformation matrix T based on the rotation matrix and translation vector. r Rewritten as a block matrix, its mathematical description is as follows:

[0115]

[0116] Step S402, based on the calculated transformation matrix T r The floating image point set is transformed as a whole to complete the pre-registration. Its mathematical description formula is:

[0117]

[0118] To complete the pre-registration.

[0119] Step S5: Use the point set shape description method to describe the initial complete and incomplete contours of the uterine fibroid. Match the point sets of the initial complete and incomplete contours of the uterine fibroid according to the shape differences and calculate the transformation matrix for fine registration.

[0120] In this embodiment, the point set shape description method is used to describe the initial complete contour of the uterine fibroid segmented by the segmentation algorithm (the complete contour segmented by the deep learning method in step S103) and the incomplete contour (the incomplete contour obtained by manual annotation in step S202). The real contour point set (that is, the complete contour obtained in step S103 and the incomplete contour obtained in step S202) is matched according to the shape difference and the transformation matrix is ​​calculated for fine registration. In this embodiment, fine registration uses an iterative nearest neighbor algorithm. Based on the pre-registration in step S4, this algorithm uses the segmented true contours and optimizes the transformation matrix iteratively by using the average distance between the two sets of points (i.e., the complete contour point set obtained in step S103 and the incomplete contour point set obtained in step S202, and the two sets of points obtained after the pre-registration step S4) as the optimization objective. This process reduces the iterative changes to a certain threshold, achieving a registered state. Specifically, this fine registration method includes: using the average Euclidean distance between the two sets of points...

[0121]

[0122] To optimize the objective, for each point pi in one set of points, let qi be the nearest neighbor of pi in another set of points. Then we get the formula for the average distance between the two sets of points. We iteratively update the spatial transformation matrix so that the change in the average distance between the sets of points is less than a certain threshold between the two iterations, thus achieving the registration state.

[0123] Exemplary apparatus

[0124] Figure 6 This is a schematic diagram of an ultrasonic image registration device provided as an exemplary embodiment of the present invention. Figure 3 As shown, the ultrasound image registration device of the present invention includes:

[0125] The complete contour segmentation and extraction unit 601 is used to acquire the patient's abdominal ultrasound image as a floating image, and extract the uterine fibroid contour in the floating image as the initial complete uterine fibroid contour based on the trained network model.

[0126] Specifically, the complete contour segmentation and extraction unit 601 further includes:

[0127] The floating image acquisition module is used to acquire abdominal ultrasound images of uterine fibroid patients with color Doppler flow imaging (CDFI) as floating images.

[0128] The floating image preprocessing module is used to perform image preprocessing on the floating image.

[0129] The deep learning network model construction and training module is used to construct a deep learning network model, divide the preprocessed floating image into a training set, a validation set, and a test set, and use the training set and the validation set to train the deep learning network model.

[0130] In this embodiment, after image preprocessing of the floating image, the delineated dataset is divided into training set, validation set and test set in a 6:2:2 ratio to train the constructed deep learning network model.

[0131] In this embodiment, the constructed deep learning network model includes two parts: a generator and a discriminator. Specifically, the ResRefineNet segmentation network is used as the generator to generate the inference results of image segmentation, and PatchGAN is used as the discriminator to determine the truth value and the generated value. The generator ResRefineNet is divided into two parts: an encoder and a decoder. The encoder uses ResNet-18 as the basic encoder, and the decoder uses ResRefineNet as the basic decoder. The encoder output and decoder input are both four layers, forming a segmentation network encoder-decoder structure. The output of each layer of the encoder is input into a basic decoder.

[0132] Specifically, the pre-processed floating image training set is input into the deep learning network model, where the generator produces the predicted image segmentation result of the uterine fibroid. Then, the generator's prediction is marked as false, and the real uterine fibroids are segmented as true. The results are input into the discriminator to train the discriminator, so that the generator and the discriminator can train against each other. The training is completed when the test set loss function reaches its minimum for the first time.

[0133] The complete contour extraction module is used to extract the contours of uterine fibroids from floating images using a trained deep learning network model.

[0134] Specifically, once the deep learning network model is trained, the unlabeled floating image is input into the trained deep learning network model to segment the uterine fibroids in the floating image and obtain the complete outline of the uterine fibroids. That is, the segmentation network in the deep learning network model is used to segment the floating image to obtain the complete outline of the uterine fibroids.

[0135] The incomplete contour acquisition unit 602 is used to acquire the corresponding abdominal grayscale ultrasound as a reference image and mark the upper boundary of the uterine fibroid to obtain the incomplete contour.

[0136] Specifically, the incomplete contour acquisition unit 602 further includes:

[0137] The reference image acquisition module is used to acquire grayscale ultrasound images of the same patient as reference images.

[0138] A reference image preprocessing module is used to preprocess the reference image.

[0139] The annotation module is used to manually annotate the outline of uterine fibroids on intraoperative guided grayscale ultrasound images, resulting in a set of incomplete outline points.

[0140] In this embodiment, the upper half of the uterine fibroid is manually annotated with incomplete outlines, where incomplete outlines refer to non-closed curved outlines. This step is mainly done manually, and the upper boundary of the uterine fibroid is marked as the incomplete outline point set.

[0141] Fitting unit 603 is used to fit the obtained complete and incomplete contours of uterine fibroids into regular geometric contours.

[0142] The complete outline of the uterine fibroid refers to the outline of the uterine fibroid in the floating image extracted by the complete outline segmentation and extraction unit 601 based on the trained network model. Specifically, in the fitting unit 603, the obtained complete and incomplete outlines of the uterine fibroid are fitted into regular geometric outlines using regular graphics. The regular graphics fitting usually uses non-rotationally symmetric graphics such as rectangles, ellipses with equal lengths and short sides or unequal major and minor axes as the regular graphics fitting for the uterine fibroid.

[0143] In a preferred embodiment, the obtained complete and incomplete contours of the uterine fibroid are fitted into regular elliptical contours P and elliptical arc contours Q using an elliptical contour fitting method, but the present invention is not limited thereto.

[0144] The pre-registration unit 604 is used to describe the regular geometric contour obtained by the fitting unit 603 using the point set shape description method, and to match the regular elliptical contour P to the regular elliptical arc contour Q according to the shape difference matching rule.

[0145] Specifically, the pre-registration unit 604 further includes:

[0146] The fitting contour shape description module is used to perform context shape description on each point in the regular graphic point set obtained by the fitting unit 603 using the shape context description method. The two points with the smallest shape difference in the two point sets form a point pair.

[0147] Specifically, the point set shape description method used in this embodiment is a shape context description method. The step of using the shape context description method to perform context shape description on each point in the regular graphic point set obtained by the fitting unit 603 further includes:

[0148] Establish a polar coordinate system with point qj in the incomplete contour point set Q of uterine fibroids as the origin, and statistically analyze the other points in the point set Q relative to point qj.j The positions are then normalized to form a statistical histogram.

[0149] Perform the same operations as in step S400a on the incomplete contour point set P of uterine fibroids (i.e., the regular elliptical contour point set) to construct a statistical histogram sequence.

[0150] Using a bipartite graph matching method, based on the sum of the histogram cost functions, we find a matching method that minimizes the total cost and perform point matching between two point sets to form point pairs.

[0151] The rigid transformation matrix calculation module is used to calculate the rigid transformation matrix between two point sets.

[0152] Specifically, the rigid transformation matrix calculation module further includes:

[0153] After establishing the complete contour point set and the incomplete contour point set, the two sets of points are decentered, as follows:

[0154]

[0155] Where P′ C With Q′ C Let P′ and Q′ be the decentralized set of paired points. and The mean of the coordinates of the paired point set.

[0156] The mathematical description of constructing a decomposition matrix is ​​as follows:

[0157] S=Q′ C P′ C T

[0158] Where S is the decomposition matrix.

[0159] The mathematical description of singular value decomposition of a matrix is ​​as follows:

[0160] S=UΣV T

[0161] Where ∑ is a diagonal matrix and U is the matrix to be selected.

[0162] The mathematical formula for calculating the rotation matrix and translation vector is as follows:

[0163]

[0164] Where R is the rotation matrix, t m This is the translation vector. The code describes the image registration method between ultrasound images.

[0165] Construct the transformation matrix T based on the rotation matrix and translation vector. r Rewritten as a block matrix, its mathematical description is as follows:

[0166]

[0167] The pre-registration implementation module is used to perform the calculation based on the transformation matrix T. r The floating image point set is transformed as a whole to complete the pre-registration. Its mathematical description formula is:

[0168]

[0169] To complete the pre-registration.

[0170] The fine registration unit 605 is used to describe the complete and incomplete contours of the initial uterine fibroid using the point set shape description method, match the point sets of the complete and incomplete contours of the initial uterine fibroid according to the shape differences, and calculate the transformation matrix to perform fine registration.

[0171] In this embodiment, the fine registration unit 605 uses a point set shape description method to describe the initial complete and incomplete contours of the uterine fibroid segmented by the segmentation algorithm. It matches the real contour point sets according to shape differences and calculates the transformation matrix for fine registration. In this embodiment, the fine registration uses an iterative nearest neighbor algorithm. This algorithm, based on the pre-registration by the pre-registration unit 604, uses the segmented real contours and takes the average distance between the two point sets as the optimization objective. It iteratively updates the transformation matrix until the iterative changes decrease to a certain threshold, achieving a registration state. Specifically, the fine registration method includes: using the average Euclidean distance between the two point sets:

[0172]

[0173] To optimize the objective, for each point pi in one set of points, let qi be the nearest neighbor of pi in another set of points. Then we get the formula for the average distance between the two sets of points. We iteratively update the spatial transformation matrix so that the change in the average distance between the sets of points is less than a certain threshold between the two iterations, thus achieving the registration state.

[0174] Exemplary electronic device

[0175] Figure 7 The structure of an electronic device provided as an exemplary embodiment of the present invention. For example... Figure 7 As shown, the electronic device includes one or more processors 71 and memory 72.

[0176] The processor 71 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0177] The memory 72 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 71 may execute the program instructions to implement the Binlog-based data synchronization method and / or other desired functions of the software programs of the various embodiments of this disclosure described above. In one example, the electronic device may also include an input device 73 and an output device 74, these components being interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0178] In addition, the input device 73 may also include, for example, a keyboard, a mouse, etc.

[0179] The output device 74 can output various information to the outside. The output device 74 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0180] Of course, for the sake of simplicity, Figure 7 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0181] Exemplary computer program product and computer readable storage medium

[0182] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the ultrasound image registration methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0183] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0184] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the ultrasound image registration methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0185] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0186] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0187] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0188] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0189] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0190] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps are decomposable and / or recombinable. Such decomposition and / or recombination should be considered equivalent to the present disclosure. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0191] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. An ultrasound image registration method, comprising the following steps: Step S1: Acquire an abdominal ultrasound image of the patient as a floating image, and extract the outline of the uterine fibroid in the floating image based on the trained network model as the initial complete outline of the uterine fibroid. Step S2: Obtain the corresponding abdominal grayscale ultrasound image as a reference image and mark the upper boundary of the uterine fibroid to obtain an incomplete outline; Step S3: Fit the obtained complete and incomplete contours of the uterine fibroids into regular geometric contours; Step S4: Use the point set shape description method to describe the regular geometric contour obtained in step S3, and match the regular elliptical contour in the regular geometric contour to the regular elliptical arc contour according to the shape difference matching rule. Step S5: Use the point set shape description method to describe the initial complete outline and incomplete outline of the uterine fibroid. Match the point set of the initial complete outline and the point set of the incomplete outline of the uterine fibroid according to the shape difference and obtain the transformation matrix for fine registration.

2. The ultrasound image registration method as described in claim 1, characterized in that, Step S1 further includes: Step S100: Acquire an abdominal ultrasound image of a patient with uterine fibroids with color Doppler blood flow imaging as a floating image; Step S101: Perform image preprocessing on the floating image; Step S102: Construct a deep learning network model by dividing the preprocessed floating image into a training set, a validation set, and a test set, and using the training set and the validation set to train the deep learning network model. Step S103: Use the trained deep learning network model to extract the outline of uterine fibroids from the floating image.

3. The ultrasound image registration method as described in claim 2, characterized in that, In step S102, the constructed deep learning network model includes a generator and a discriminator, and its training or inference process includes: The generator in the deep learning network model contains an encoder that encodes the network input to obtain a feature map; The generator in the deep learning network model includes a decoder that decodes the feature map to obtain the segmentation inference result as the generator output. The pre-processed and labeled true segments of uterine fibroids are input into the discriminator along with the generator output. The discriminator and generator are trained simultaneously, making the generator compete against the discriminator, thereby enabling the generator to be used.

4. The ultrasound image registration method as described in claim 3, characterized in that, Step S2 further includes: Step S200: Obtain a grayscale ultrasound image of the same patient as a reference image; Step S201: Preprocess the reference image; Step S202: Manually annotate the outline of the uterine fibroid on the intraoperative guided grayscale ultrasound image to obtain an incomplete outline point set.

5. The ultrasound image registration method as described in claim 4, characterized in that, In step S3, the obtained complete and incomplete contours of the uterine fibroid are fitted into the regular elliptical contour and the regular elliptical arc contour using the elliptical contour fitting method.

6. The ultrasound image registration method as described in claim 5, characterized in that, Step S4 further includes: Step S400: Use the shape context description method to perform context shape description on each point in the regular graphic point set obtained in step S3, and form a point pair with the two points with the smallest shape difference between the two point sets. Step S401: Calculate the rigid transformation matrix between the two point sets; Step S402: The floating image point set is transformed as a whole according to the calculated transformation matrix to complete the pre-registration.

7. The ultrasound image registration method as described in claim 6, characterized in that, Step S400 further includes: Step S400a, for a point q in the set of points of the regular elliptical arc contour... j Establish a polar coordinate system with point q as the origin, and statistically analyze the other points in the elliptical arc contour of the aforementioned rule relative to point q. j The positions are then normalized to form a statistical histogram. Step S400b: Perform the same operation as in step S400a on the set of elliptical contour points of the rule to calculate and construct a statistical histogram sequence. Step S400c: Using the bipartite graph matching method, with the sum of the histogram cost functions as the standard, find the matching method that minimizes the total cost, and perform point set matching on the two point sets to form point pairs.

8. The ultrasound image registration method as described in claim 7, characterized in that, In step S5, the fine registration uses the iterative nearest algorithm. Based on the pre-registration in step S4, the real contours segmented in step S1 are used, and the average distance between the two sets of points is used as the optimization objective. The transformation matrix is ​​continuously updated iteratively until the iterative change is reduced to a certain threshold, thus achieving the registration state.

9. An ultrasound image registration device, comprising: The complete contour segmentation and extraction unit is used to acquire the patient's abdominal ultrasound image as a floating image, and extract the uterine fibroid contour in the floating image as the initial complete uterine fibroid contour based on the trained network model. The incomplete contour acquisition unit is used to acquire the corresponding abdominal grayscale ultrasound as a reference image and mark the upper boundary of the uterine fibroid to obtain the incomplete contour. The fitting unit is used to fit the obtained complete and incomplete contours of uterine fibroids into regular geometric contours. The pre-registration unit is used to describe the regular geometric contour obtained by the fitting unit using the point set shape description method, and to match the regular elliptical contour in the regular geometric contour to the regular elliptical arc contour according to the shape difference matching rule. The fine registration unit is used to describe the initial complete and incomplete contours of the uterine fibroid using a point set shape description method. It matches the point sets of the initial complete and incomplete contours of the uterine fibroid according to the shape differences and calculates the transformation matrix for fine registration.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the ultrasound image registration method as described in any one of claims 1 to 8.

Images