Image registration method, device, computer equipment and storage medium

Abstract

The application relates to an image registration method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a real-time ultrasound image of a target object and a stored target image; using a feature point detection algorithm to extract a first feature point set in the real-time ultrasound image and a second feature point set in the stored target image; determining a transformation function based on a transformation matrix and the coordinates of the feature points in a target feature point pair; updating the transformation matrix so that the function value of the transformation function changes correspondingly; continuously updating the transformation matrix until the function value converges; determining the transformation matrix corresponding to the function value convergence as a rigid transformation matrix; converting the coordinates of the image lesion area based on the rigid transformation matrix to obtain a target lesion area; and fusing the target lesion area with the real-time ultrasound image to obtain a registered ultrasound image. The method can improve the accuracy of the registration result.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to an image registration method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of medical device technology, puncture localization in puncture surgery is often achieved by image registration of MRI and ultrasound images with the lesion area marked. Since the target object may move during the puncture, puncture errors can occur. To compensate for these errors, existing methods involve manual or semi-automatic adjustment of the registration results. However, image registration methods that rely on manual or semi-automatic adjustments suffer from low registration accuracy. Summary of the Invention

[0003] Therefore, it is necessary to address the problem of low registration accuracy in existing image registration methods by providing an image registration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of registration results.

[0004] Firstly, this application provides an image registration method. The method includes:

[0005] Acquire real-time ultrasound images of the target object and existing target images, including the lesion region in the image;

[0006] A feature point detection algorithm is used to extract the first set of feature points from the real-time ultrasound image and the second set of feature points from the stored target image, respectively.

[0007] Based on the first feature point set and the second feature point set, a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set is determined.

[0008] Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, the transformation function is determined. The transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix that makes the function value converge is determined as the rigid transformation matrix.

[0009] The coordinates of the lesion region in the image are transformed using a rigid transformation matrix to obtain the target lesion region; the target lesion region is then fused with the real-time ultrasound image to obtain the registered ultrasound image.

[0010] Secondly, this application also provides an image registration apparatus. The apparatus includes:

[0011] The image acquisition module is used to acquire real-time ultrasound images of the target object and stored target images, including the lesion area in the image;

[0012] The feature point extraction module is used to extract the first set of feature points from the real-time ultrasound image and the second set of feature points from the stored target image using a feature point detection algorithm.

[0013] The feature point pair determination module is used to determine a target feature point pair composed of feature points in the first feature point set and feature points in the second feature point set, based on a first feature point set and a second feature point set.

[0014] The rigid transformation matrix determination module is used to determine the transformation function based on the transformation matrix and the coordinates of the feature points in the target feature point pair. The transformation matrix is ​​updated to make the function value of the transformation function change accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix corresponding to the convergence of the function value is determined as the rigid transformation matrix.

[0015] The fusion module is used to transform the coordinates of the lesion region in the image based on a rigid transformation matrix to obtain the target lesion region; and to fuse the target lesion region with the real-time ultrasound image to obtain the registered ultrasound image.

[0016] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0017] Acquire real-time ultrasound images of the target object and existing target images, including the lesion region in the image;

[0018] A feature point detection algorithm is used to extract the first set of feature points from the real-time ultrasound image and the second set of feature points from the stored target image, respectively.

[0019] Based on the first feature point set and the second feature point set, a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set is determined.

[0020] Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, the transformation function is determined. The transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix that makes the function value converge is determined as the rigid transformation matrix.

[0021] The coordinates of the lesion region in the image are transformed using a rigid transformation matrix to obtain the target lesion region; the target lesion region is then fused with the real-time ultrasound image to obtain the registered ultrasound image.

[0022] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0023] Acquire real-time ultrasound images of the target object and existing target images, including the lesion region in the image;

[0024] A feature point detection algorithm is used to extract the first set of feature points from the real-time ultrasound image and the second set of feature points from the stored target image, respectively.

[0025] Based on the first feature point set and the second feature point set, a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set is determined.

[0026] Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, the transformation function is determined. The transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix that makes the function value converge is determined as the rigid transformation matrix.

[0027] The coordinates of the lesion region in the image are transformed using a rigid transformation matrix to obtain the target lesion region; the target lesion region is then fused with the real-time ultrasound image to obtain the registered ultrasound image.

[0028] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0029] Acquire real-time ultrasound images of the target object and existing target images, including the lesion region in the image;

[0030] A feature point detection algorithm is used to extract the first set of feature points from the real-time ultrasound image and the second set of feature points from the stored target image, respectively.

[0031] Based on the first feature point set and the second feature point set, a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set is determined.

[0032] Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, the transformation function is determined. The transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix that makes the function value converge is determined as the rigid transformation matrix.

[0033] The coordinates of the lesion region in the image are transformed using a rigid transformation matrix to obtain the target lesion region; the target lesion region is then fused with the real-time ultrasound image to obtain the registered ultrasound image.

[0034] The aforementioned image registration method, apparatus, computer equipment, storage medium, and computer program product, by acquiring real-time ultrasound images of the target object and stored target images, and employing a feature point detection algorithm to extract a first set of feature points from the real-time ultrasound image and a second set of feature points from the stored target image, can improve the accuracy of the extracted feature points, thus improving the precision of the image registration results. The target feature point pair consists of feature points from the first and second feature point sets. Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, a transformation function is determined. The transformation matrix is ​​updated to change the function value accordingly. This process continues until the function value converges, and the transformation matrix corresponding to the convergence of the function value is determined as the rigid transformation matrix. The coordinates of the image lesion region are transformed based on the rigid transformation matrix to obtain the target lesion region. Since the existing target image includes the lesion region, this method of transforming the coordinates of the lesion region by making the transformation function converge to the corresponding transformation matrix can ensure that the obtained target lesion region is closer to the real lesion region. By fusing the target lesion region with the real-time ultrasound image, the obtained registered ultrasound image has high accuracy and improves the precision of the registration result. Attached Figure Description

[0035] Figure 1 This is an application environment diagram of the image registration method in one embodiment;

[0036] Figure 2 This is a flowchart illustrating an image registration method in one embodiment;

[0037] Figure 3 This is a schematic diagram of a sub-process of S203 in one embodiment;

[0038] Figure 4 This is a schematic diagram of a sub-process of S204 in one embodiment;

[0039] Figure 5 This is a schematic diagram of a sub-process of S201 in one embodiment;

[0040] Figure 6 This is a schematic diagram of a sub-process of S506 in one embodiment;

[0041] Figure 7 This is a schematic diagram of a sub-process of S608 in one embodiment;

[0042] Figure 8 This is a schematic diagram of a sub-process of S201 in another embodiment;

[0043] Figure 9 This is a schematic diagram illustrating the prerequisites for an image registration method in one embodiment;

[0044] Figure 10 This is a schematic diagram of the overall process of the image registration method in one embodiment;

[0045] Figure 11 This is a schematic diagram of a model registration method in one embodiment;

[0046] Figure 12 This is a schematic diagram illustrating the registration of a real-time ultrasound image and a stored target image in one embodiment;

[0047] Figure 13 This is a schematic diagram of corner points in an image from one embodiment;

[0048] Figure 14 This is a schematic diagram of feature points in a real-time ultrasound image in one embodiment.

[0049] Figure 15 This is a schematic diagram illustrating the determination of target feature point pairs in one embodiment;

[0050] Figure 16 This is a schematic diagram illustrating the determination of a first target region and a second target region in one embodiment;

[0051] Figure 17 This is a schematic diagram illustrating the determination of target feature point pairs in yet another embodiment;

[0052] Figure 18 This is a schematic diagram illustrating the assignment of different weights to pairs of target feature points in one embodiment;

[0053] Figure 19 This is a schematic diagram illustrating coordinate transformation using a rigid transformation matrix in one embodiment;

[0054] Figure 20 This is a schematic diagram illustrating the puncture position change in one embodiment;

[0055] Figure 21 This is a structural block diagram of an image registration device in one embodiment;

[0056] Figure 22 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The image registration method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. The image registration method provided in this application embodiment can be executed by terminal 102 or server 104 alone, or by terminal 102 and server 104 in cooperation. Taking the execution by terminal 102 alone as an example: terminal 102 acquires a real-time ultrasound image of the target object and a stored target image, the stored target image including the image lesion region; a feature point detection algorithm is used to extract a first feature point set from the real-time ultrasound image and a second feature point set from the stored target image; based on the first and second feature point sets, a target feature point pair composed of feature points from the first and second feature point sets is determined; based on the transformation matrix and the feature point coordinates in the target feature point pair, a transformation function is determined, and the transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges, and the transformation matrix corresponding to the convergence of the function value is determined as a rigid transformation matrix; the coordinates of the image lesion region are transformed based on the rigid transformation matrix to obtain the target lesion region; the target lesion region is fused with the real-time ultrasound image to obtain the registered ultrasound image. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0059] In one embodiment, such as Figure 2 As shown, an image registration method is provided, which can be applied to a computer device (the computer device can be...) Figure 1 Taking terminal 102 or server 104 as an example, the following steps are included:

[0060] S201, acquire real-time ultrasound images of the target object and existing target images, the existing target images including the lesion area in the image.

[0061] The target object is the object to be punctured. The computer device acquires real-time ultrasound images of the target cross-section and a previously stored target image. A cross-section is a section perpendicular to the axis of the target object. For example, the target cross-section refers to the cross-section between the top and bottom of the target object. The computer device acquires real-time ultrasound images of the target cross-section using an ultrasound detection device.

[0062] The stored target image is an ultrasound image of a target cross-section stored in a computer device, including an image lesion region. The image lesion region is used to indicate the lesion area of ​​the target object in the stored target image. In some embodiments, the image lesion region is a delineation in the stored target image.

[0063] S202 uses a feature point detection algorithm to extract the first feature point set from the real-time ultrasound image and the second feature point set from the stored target image.

[0064] Feature point detection algorithms are used to detect feature points in an image. Feature points can be corners or points with significant gray-level gradient changes. Corners are points with drastic brightness changes or intersections of two lines, and are important image features. Optionally, the feature point detection algorithm can be the Moravec feature point detection algorithm or the Harris corner feature point detection algorithm (a feature point detection algorithm proposed by Chris Harris and Mike Stephens).

[0065] The computer equipment uses a feature point detection algorithm to detect feature points in real-time ultrasound images, obtaining a first feature point set, which includes at least one first feature point. The computer equipment then uses the same algorithm to detect feature points in a stored target image, obtaining a second feature point set, which includes at least one second feature point. Because image feature points are stable, using the feature point detection algorithm to extract feature points from both real-time ultrasound images and stored target images offers high accuracy.

[0066] S203, based on the first feature point set and the second feature point set, determine the target feature point pair composed of feature points in the first feature point set and feature points in the second feature point set.

[0067] In this context, a target feature point pair consists of feature points from a first feature point set and feature points from a second feature point set. Each target feature point pair contains two feature points. Since both real-time ultrasound images and stored target images are ultrasound images of the target's cross-section, they share a certain degree of similarity.

[0068] A computer device determines a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set, based on a first feature point set and a second feature point set. In some embodiments, for each first feature point in the first feature point set, the computer device forms a target feature point pair with each second feature point in the second feature point set. In other embodiments, the computer device determines the similarity between each first feature point in the first feature point set and each second feature point in the second feature point set, and forms a target feature point pair with feature points in the first feature point set and feature points in the second feature point set that have a higher similarity.

[0069] S204. Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, determine the transformation function. Update the transformation matrix to make the function value of the transformation function change accordingly. Continue to update the transformation matrix until the function value converges. The transformation matrix that makes the function value converge is determined as the rigid transformation matrix.

[0070] The transformation function is used for image registration. The computer device inputs the transformation matrix and the coordinates of the feature points in each target feature point pair into the initial transformation function to determine the transformation function. The transformation matrix is ​​the parameter in the transformation function. The computer device updates the transformation matrix to change the function value accordingly. The computer device continues to update the transformation matrix until the function value converges. The computer device determines the transformation matrix that causes the function value to converge as the rigid transformation matrix. The rigid transformation matrix is ​​used to transform the coordinates of the lesion region in the image.

[0071] 205. The coordinates of the lesion region in the image are transformed based on the rigid transformation matrix to obtain the target lesion region; the target lesion region is fused with the real-time ultrasound image to obtain the registered ultrasound image.

[0072] In this process, the computer device uses a rigid transformation matrix to transform the lesion region in the image. For example, the computer device multiplies the coordinates of the lesion region in the image by the rigid transformation matrix to obtain the transformed coordinates. The computer device then uses the region determined based on the transformed coordinates as the target lesion region. In some embodiments, the rigid transformation matrix is ​​a composite matrix composed of rotation and translation matrices. The target lesion region obtained by transforming the coordinates of the lesion region in the image using the rigid transformation matrix has the same shape as the lesion region in the image. The computer device then fuses the target lesion region with the real-time ultrasound image. Specifically, the computer device delineates the target lesion region in the real-time ultrasound image based on its coordinates, resulting in the delineated image, which is the registered ultrasound image. The registered ultrasound image includes the target lesion region.

[0073] In the aforementioned image registration method, by acquiring the real-time ultrasound image of the target object and the existing target image, a feature point detection algorithm is used to extract the first feature point set from the real-time ultrasound image and the second feature point set from the existing target image, respectively. This improves the accuracy of the extracted feature points and enhances the precision of the image registration result. The target feature point pair consists of feature points from the first feature point set and the second feature point set. Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, a transformation function is determined. The transformation matrix is ​​updated to change the function value accordingly. This process is continued until the function value converges, and the transformation matrix corresponding to the convergence of the function value is defined as the rigid transformation matrix. The coordinates of the image lesion region are transformed based on the rigid transformation matrix to obtain the target lesion region. Since the existing target image includes the image lesion region, this method of using the transformation matrix corresponding to the convergence of the transformation function to transform the coordinates of the image lesion region ensures that the obtained target lesion region is closer to the real lesion region. By fusing the target lesion region with the real-time ultrasound image, the resulting registered ultrasound image has high accuracy, thus improving the precision of the registration result.

[0074] In one embodiment, such as Figure 3 As shown, the first feature point set includes at least one first feature point, and the second feature point set includes at least one second feature point; based on the first and second feature point sets, a target feature point pair composed of feature points from the first and second feature point sets is determined, including:

[0075] S302, for each first feature point in the first feature point set, determine a first target region in the real-time ultrasound image containing the current first feature point, and acquire a preset number of first pixels in the first target region.

[0076] The first feature point set includes at least one first feature point. For each first feature point in the first feature point set, the computer device determines a first target region in the real-time ultrasound image that contains the current first feature point. The first target region is an image region in the real-time ultrasound image that contains the current first feature point, and the first target region includes a preset number of first pixels. Exemplarily, the first target region can be a square region, a circular region, or a region of any shape containing the current first feature point. Optionally, the first target region can also be a square region, a circular region, or a region of any shape centered on the current first feature point. The computer device acquires a preset number of first pixels in the first target region.

[0077] S304, for each second feature point in the second feature point set, determine the second target region in the stored target image that contains the current second feature point, and obtain a preset number of second pixels in the second target region.

[0078] The second feature point set includes at least one second feature point. For each second feature point in the second feature point set, the computer device determines a second target region in the stored target image that contains the current second feature point. The second target region is an image region in the stored target image that contains the current second feature point, and the second target region includes a preset number of second pixels. The second target region is the same size and shape as the first target region. For example, the second target region can be a square region, a circular region, or a region of any shape containing the current second feature point. Optionally, the second target region can also be a square region, a circular region, or a region of any shape centered on the current second feature point. The computer device acquires a preset number of second pixels in the second target region.

[0079] S306, for each first target region in each first target region, based on the pixel value of the first pixel in the current first target region and the pixel value of the second pixel in each second target region, determine the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region; if the similarity between the first target feature point and any second target feature point is greater than a preset value, then the first target feature point and the corresponding second target feature point are determined as a target feature point pair.

[0080] Among them, the first target feature point is the first feature point corresponding to the current first target region, and the second target feature point is the second feature point corresponding to the second target region.

[0081] For each first target region, based on the pixel value of the first pixel in the current first target region and the pixel value of the second pixel in each second target region, the computer device determines the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region. The computer device compares the similarity between the first target feature point in the current first target region and the second target feature point in any second target region with a preset value. If the similarity between the first target feature point and any second target feature point is greater than the preset value, then the first target feature point and the corresponding second target feature point are identified as a target feature point pair. This method of using feature points and the pixel values ​​of pixels containing the feature point region to jointly determine the similarity of feature points has high accuracy, and identifying pairs with high similarity as target feature point pairs helps improve the accuracy of the registration results.

[0082] In this embodiment, by determining a first target region containing the current first feature point in a real-time ultrasound image and a second target region containing the current second feature point in a stored target image, and based on the first pixel in the first target region and a preset number of second pixels in the second target region, the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region is determined. The first target feature point and the corresponding second target feature point with a similarity greater than a preset value are determined as a target feature point pair. The determined target feature point pair is a feature point with a high degree of similarity, which can eliminate the interference of feature points with a low degree of similarity, and is conducive to improving the accuracy of image registration results.

[0083] In one embodiment, determining the similarity between a first target feature point corresponding to the current first target region and a second target feature point corresponding to each second target region, based on the pixel value of a first pixel point in the current first target region and the pixel value of a second pixel point in each second target region, includes: for each second target region, obtaining a first difference between the pixel value of each first pixel point in the current first target region and the pixel value of each second pixel point in the current second target region; obtaining a first pixel average value in the current first target region and a second difference between the first pixel average value in the current second target region; obtaining a histogram similarity between a first pixel histogram in the current first target region and a second pixel histogram in the current second target region; using the sum of squares of each first difference as a first pixel difference; using the sum of squares of each second difference as a second pixel difference; using the histogram similarity as a third pixel difference; and determining the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region based on the first pixel difference, the second pixel difference, and the third pixel difference.

[0084] Specifically, for each second target region, the computer device acquires a first difference between the pixel value of each first pixel in the current first target region and the pixel value of each second pixel in the current second target region, and uses the sum of the squares of these first differences as the first pixel difference. The computer device also acquires a second difference between the average first pixel value of the current first target region and the average second pixel value of the current second target region, and uses the sum of the squares of these second differences as the second pixel difference. Furthermore, the computer device acquires the histogram similarity between the histogram of the first pixel in the current first target region and the histogram of the second pixel in the current second target region, and uses this histogram similarity as the third pixel difference. Based on the first pixel difference, the second pixel difference, and the third pixel difference, the computer device determines the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region.

[0085] For example, the similarity can be calculated as follows: D = α1*d1 + α2*d2 - α3*d3. Here, D represents the similarity between the first target feature point and any second feature point. d1 represents the first pixel difference, representing the sum of squares of the differences between the pixel values ​​of each first pixel in the first target region and the pixel values ​​of each second pixel in the second target region. d2 represents the second pixel difference, representing the sum of squares of the differences between the average pixel value of all first pixels in the first target region and the average pixel value of all second pixels in the second target region. d3 represents the third pixel difference, representing the similarity between the first pixel histogram of the first target region and the second pixel histogram of the second target region. a1, a2, and -a3 are the weights of d1, d2, and d3, respectively. The values ​​of a1, a2, and -a3 can be flexibly selected according to the actual situation. The smaller the value of D, the higher the similarity between the two feature points.

[0086] Specifically, d1 can be calculated as follows: For each first pixel in the first target region, the difference between the current first pixel value and the pixel value of each second pixel is calculated. The squares of each difference are then summed, and the sum is d1. d3 can be calculated as follows: Based on the pixel values ​​of the first pixels in the first target region, a first pixel histogram is determined; based on the pixel values ​​of the second pixels in the second target region, a second pixel histogram is determined; a histogram similarity algorithm is used to determine the similarity between the first and second pixel histograms, and the resulting similarity is d3.

[0087] In some embodiments, the similarity can be calculated using any combination of α1*d1, α2*d2, and -α3*d3. For example, D = α1*d1, D = α2*d2, D = α1*d1 + α2*d2, or D = α1*d1 - α3*d3.

[0088] In this embodiment, the similarity between the first target feature point and the second feature point is determined based on the pixel values ​​of the corresponding first target region and the corresponding second target region. This similarity calculation method combines the information of the pixels in the region containing the feature point, which is beneficial to obtaining accurate similarity determination results.

[0089] In one embodiment, such as Figure 4 As shown, based on the transformation matrix and the coordinates of the feature points in the target feature point pair, the transformation function is determined, including:

[0090] S402, for each target feature point pair in the target feature point pair, based on the coordinates of the current target feature point pair, determine whether the current target feature point pair is within the lesion area of ​​the image; if it is, reset the weight corresponding to the current target feature point pair to the first preset value; if it is not, reset the weight corresponding to the current target feature point pair to the second preset value; wherein, the first preset value is greater than the second preset value.

[0091] Specifically, for each target feature point pair, the computer device determines whether the current target feature point pair is within the lesion area of ​​the image based on the coordinates of the current target feature point pair.

[0092] If either the coordinates of the first feature point or the second feature point in the current target feature point pair is less than the coordinates of the boundary of the image lesion region, it means that the coordinates of the current target feature point pair are within the image lesion region, and the computer device resets the weight corresponding to the current target feature point pair to the first preset value.

[0093] If the coordinates of the first and second feature points in the current target feature point pair are not less than the coordinates of the boundary of the image lesion region, it indicates that the coordinates of the current target feature point pair are not within the image lesion region. The computer device then resets the weights corresponding to the current target feature point pair to a second preset value. The first preset value is greater than the second preset value, ensuring that the weights of target feature point pairs within the image lesion region are greater than those outside the image lesion region. This improves the effectiveness of target feature point pairs within the image lesion region and enhances the accuracy of image registration results.

[0094] S404, based on the weights and transformation matrices of each target feature point pair, determines the objective function corresponding to each target feature point pair.

[0095] The computer device inputs the weights and transformation matrix corresponding to each target feature point pair into the initial transformation function to obtain the target function corresponding to each target feature point pair. For example, the initial transformation function is: F i =w i ||q i -Rp i -t||, where q i p represents the first feature point in a real-time ultrasound image. i Let R represent the second feature point in the existing target image, t represent the rotation matrix, and w represent the translation matrix. The transformation matrix is ​​a composite matrix of R and t. i F represents the weights corresponding to each target feature point pair. i This represents the objective function corresponding to each target feature point.

[0096] S406, sum the corresponding objective functions for each target feature point to obtain the transformation function.

[0097] The computer device sums the values ​​of each target feature point with respect to the corresponding objective function to obtain the transformation function. For example, the transformation function F is: F = Σw i ||q i -Rp i -t||.

[0098] In this embodiment, by determining whether each target feature point pair is within the lesion region of the image, and based on the weights and transformation matrix of each target feature point pair, the objective function corresponding to each target feature point pair is determined, thereby obtaining the transformation function. Since the weights of target feature point pairs within the lesion region are greater than those outside the lesion region, this is beneficial to improving the effect of the target feature point pairs within the lesion region on the transformation function, thus improving the accuracy of the image registration results.

[0099] In one embodiment, such as Figure 5 As shown, the steps for acquiring a stored target image include:

[0100] S502, acquire the initial ultrasound image and initial image of the target object.

[0101] The initial ultrasound image of the target object can be an initial ultrasound image of a target cross-section within the target object. The initial image of the target object can be an initial image of a target cross-section within the target object. Before acquiring real-time ultrasound images, the computer device acquires the initial ultrasound image of the target cross-section and the initial image of the target cross-section within the target object. The initial ultrasound image is an ultrasound image, and the initial image can be a medical image; for example, the initial image can be either an MRI (Magnetic Resonance Imaging) image or a CT (Computed Tomography) image.

[0102] S504, Perform three-dimensional reconstruction on the initial ultrasound image to obtain the ultrasound three-dimensional model of the target object; Perform three-dimensional reconstruction on the initial image to obtain the image three-dimensional model of the target object.

[0103] Three-dimensional reconstruction refers to the method of establishing a mathematical model of a three-dimensional object that is suitable for computer representation and processing. The initial ultrasound image includes at least one ultrasound image, and the initial imaging includes at least one medical image.

[0104] Computer equipment performs 3D reconstruction on the initial ultrasound image to obtain a 3D ultrasound model of the target object. The computer equipment performs 3D reconstruction on the initial image to obtain a 3D image model of the target object.

[0105] S506, register the ultrasound 3D model and the imaging 3D model to obtain the stored target image.

[0106] In this process, computer equipment registers the ultrasound 3D model and the image 3D model to obtain the stored target image. Specifically, the registration method combines flexible and rigid registration. The registration of the ultrasound 3D model and the image 3D model by computer equipment is performed before acquiring the real-time ultrasound image, which helps improve registration efficiency.

[0107] In this embodiment, a three-dimensional ultrasound model of the target object is obtained by performing three-dimensional reconstruction on the acquired initial ultrasound image, and an image three-dimensional model of the target object is obtained by performing three-dimensional reconstruction on the acquired initial image. Since the initial image has clearer image details than the ultrasound image, registering the ultrasound three-dimensional model and the image three-dimensional model can obtain the existing target image, which is beneficial to improving the accuracy of the existing target image.

[0108] In one embodiment, such as Figure 6 As shown, the 3D model of the target object includes the lesion region of the model; the ultrasound 3D model and the image 3D model are registered to obtain the existing target image, including:

[0109] S602, based on the ultrasound 3D model and the image 3D model, performs bounding box registration to obtain the initial linear transformation matrix.

[0110] Bounding box registration is a method for registering the ultrasound 3D model and the image 3D model. Optionally, the bounding box registration method used can be OBB (Oriented Bounding Box) registration. The computer equipment obtains the initial linear transformation matrix by performing bounding box registration on the ultrasound 3D model and the image 3D model.

[0111] S604. The initial linear transformation matrix is ​​used to perform coordinate transformation on the three-dimensional model of the image to obtain the first transformation model; based on the ultrasound three-dimensional model and the first transformation model, iterative nearest neighbor registration is performed to obtain the linear transformation matrix.

[0112] In this process, the computer equipment uses an initial linear transformation matrix to perform coordinate transformation on the image 3D model, obtaining a first transformed model. For example, the coordinates of each point in the image 3D model can be multiplied by the initial linear transformation matrix to obtain the transformed coordinates of each point, thereby determining the first transformed model. In some embodiments, the initial linear transformation matrix is ​​a composite matrix of rotation and translation matrices, and the first transformed model has the same shape as the image 3D model. After bounding box registration and coordinate transformation using the initial linear transformation matrix, the resulting first transformed model's position in space is relatively close to its position in space within the ultrasound 3D model.

[0113] Iterative Closest Point (ICP) registration is a method for registering an ultrasound 3D model and a first transformation model. The computer device obtains a linear transformation matrix by performing iterative nearest neighbor registration on the ultrasound 3D model and the first transformation model.

[0114] S606, the first transformation model is transformed by a linear transformation matrix to obtain the second transformation model; based on the ultrasonic three-dimensional model, the second transformation model is elastically registered by a thin plate spline algorithm to obtain a nonlinear transformation matrix.

[0115] In this process, the computer device performs coordinate transformation on the first transformation model using a linear transformation matrix to obtain a second transformation model. For example, the coordinates of each point in the first transformation model can be multiplied by the linear transformation matrix to obtain the transformed coordinates of each point, thereby determining the second transformation model. In some embodiments, the linear transformation matrix is ​​a composite matrix of rotation and translation matrices, and the second transformation model has the same shape as the first transformation model. After iterative nearest neighbor registration and coordinate transformation using the linear transformation matrix, the spatial position of the obtained second transformation model is closer to the spatial position of the ultrasonic 3D model than the spatial position of the first transformation model.

[0116] The Thin Plate Spline (TPS) algorithm is a non-rigid deformation algorithm. Based on the ultrasonic 3D model, the computer equipment uses the TPS algorithm to elastically register the second transformed model, obtaining a nonlinear transformation matrix. Using the nonlinear transformation matrix allows the second transformed model to deform, resulting in a second transformed model that more closely resembles the ultrasonic 3D model.

[0117] S608, based on the initial linear transformation matrix, the linear transformation matrix, the nonlinear transformation matrix, the model lesion region, and the ultrasound three-dimensional model, obtains the existing target image.

[0118] The computer equipment transforms the lesion area of ​​the model based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix, and combines the ultrasound three-dimensional model to obtain the stored target image.

[0119] In this embodiment, bounding box registration is performed using the ultrasound 3D model and the image 3D model to obtain an initial linear transformation matrix. Iterative nearest neighbor registration is then performed based on the ultrasound 3D model and the first transformation model to obtain another linear transformation matrix. Based on the ultrasound 3D model, a thin-plate spline algorithm is used to elastically register the second transformation model to obtain a nonlinear transformation matrix. Multiple transformation matrices are used to transform the model's lesion region, resulting in a transformed lesion region that more closely approximates the lesion region of the initial ultrasound image. Since the image 3D model of the target object includes the model's lesion region, the obtained stored target image will include the image's lesion region. Image registration based on the stored target image helps improve the accuracy of the image registration results.

[0120] In one embodiment, such as Figure 7 As shown, based on the initial linear transformation matrix, the linear transformation matrix, the nonlinear transformation matrix, the model lesion region, and the ultrasound 3D model, the existing target image is obtained, including:

[0121] S702, based on the initial linear transformation matrix, the linear transformation matrix and the nonlinear transformation matrix, performs coordinate transformation on the lesion region of the model to obtain the transformed lesion region.

[0122] The computer equipment multiplies the coordinates of the model lesion region by an initial linear transformation matrix, a subsequent linear transformation matrix, and a nonlinear transformation matrix to achieve coordinate transformation of the model lesion region. Since the lesion region in the initial ultrasound image is unclear, compared to directly mapping the model lesion region to the ultrasound 3D model, the transformed lesion region will more closely approximate the actual location and shape of the lesion region in the ultrasound 3D model.

[0123] S704 fuses the converted lesion area with the ultrasound 3D model to obtain the fused ultrasound model.

[0124] The computer equipment fuses the converted lesion area with the 3D ultrasound model to obtain a fused ultrasound model. Specifically, the computer equipment outlines the lesion area in the 3D ultrasound model based on the coordinates of the converted lesion area, and the outlined ultrasound model is the fused ultrasound model. The fused ultrasound model includes the outlined lesion area.

[0125] S706, determine the ultrasound image corresponding to the target object from the fused ultrasound model, and use the ultrasound image as the stored target image.

[0126] In this process, the computer equipment determines the ultrasound image corresponding to the target cross-section in the target object from the fused ultrasound model, and uses the ultrasound image corresponding to the target cross-section in the fused ultrasound model as the stored target image. The stored target image includes the lesion region in the image.

[0127] In this embodiment, multiple transformation matrices are used to obtain the transformed lesion region, which approximates the actual lesion region location and shape in the ultrasound 3D model. The transformed lesion region is then fused with the ultrasound 3D model to obtain a fused ultrasound model. The existing target image is then determined from the fused ultrasound model. The existing target image has a more accurate lesion region than the initial ultrasound image. Image registration based on the existing target image will improve the accuracy of the image registration results.

[0128] In one embodiment, such as Figure 8 As shown, the steps for acquiring the stored target image also include:

[0129] S801, acquire the initial ultrasound image of the target object and the initial image of the target object; the initial image includes the lesion area.

[0130] S802, based on the initial ultrasound image and initial image, performs bounding box registration to obtain the initial linear transformation matrix.

[0131] S803, the initial image is transformed using an initial linear transformation matrix to obtain the first transformed image; based on the initial ultrasound image and the first transformed image, iterative nearest neighbor registration is performed to obtain the linear transformation matrix.

[0132] S804, the first transformed image is transformed by a linear transformation matrix to obtain the second transformed image; based on the initial ultrasound image, the second transformed image is elastically registered by a thin plate spline algorithm to obtain a nonlinear transformation matrix.

[0133] S805 performs coordinate transformation on the image lesion region based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix to obtain the transformed lesion region; the transformed lesion region is then fused with the initial ultrasound image to obtain the stored target image.

[0134] The process involves a computer device acquiring an initial ultrasound image and an initial image of the target cross-section of the target object, the initial image including the lesion region. Boundary box registration is performed based on the initial ultrasound image and the initial image to obtain an initial linear transformation matrix. This initial linear transformation matrix is ​​then used to perform coordinate transformation on the initial image to obtain a first transformed image. In some embodiments, the initial linear transformation matrix is ​​a composite matrix of rotation and translation, and the shape of the lesion region in the first transformed image is the same as that in the initial image. After boundary box registration and coordinate transformation using the initial linear transformation matrix, the position of the lesion region in the obtained first transformed image is relatively close to the position of the lesion region in the initial ultrasound image.

[0135] The computer device performs iterative nearest neighbor registration based on the initial ultrasound image and the first transformed image to obtain a linear transformation matrix. This linear transformation matrix is ​​then used to perform coordinate transformation on the first transformed image to obtain a second transformed image. In some embodiments, the linear transformation matrix is ​​a composite matrix of rotation and translation, and the shape of the lesion region in the second transformed image is the same as that in the first transformed image. After iterative nearest neighbor registration and coordinate transformation using the linear transformation matrix, the position of the lesion region in the second transformed image is closer to the position of the lesion region in the initial ultrasound image than the position of the lesion region in the first transformed image.

[0136] The computer equipment uses a thin-plate spline algorithm to elastically register the second transformed image based on the initial ultrasound image, obtaining a nonlinear transformation matrix. Based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix, the coordinates of the lesion region in the image are transformed to obtain the transformed lesion region. The transformed lesion region is then fused with the initial ultrasound image to obtain the stored target image. Specifically, the computer equipment delineates the lesion region in the initial ultrasound image according to the coordinates of the transformed lesion region; the delineated ultrasound image is the stored target image. The stored target image includes the image lesion region.

[0137] In this embodiment, multiple registrations are used to obtain multiple transformation matrices. The transformed lesion region obtained based on these multiple transformation matrices closely approximates the actual lesion region location and shape in the initial ultrasound image. The transformed lesion region is then fused with the initial ultrasound image to obtain a stored target image. The stored target image has a more accurate image lesion region than the initial ultrasound image. Image registration based on the stored target image will help improve the accuracy of the image registration results.

[0138] To illustrate the image registration method and its effects in this scheme in detail, a specific embodiment is described below:

[0139] For the application of puncture targeting the target object, the image registration method proposed in this application is based on the following premises: the target object has been displaced before each image registration and has stopped moving. After image registration is completed, puncture is performed using the lesion area indicated by the image registration; the target object does not undergo significant deformation during the image registration process. Figure 9 The diagram shows the prerequisites for the image registration method.

[0140] The computer equipment acquires real-time ultrasound images of the target cross-section and previously stored target images, including lesion areas in the images. For example... Figure 10 The diagram shows the overall process flow of the image registration method.

[0141] There are two methods for acquiring the existing target image.

[0142] The first method is the model registration method, such as... Figure 11 The diagram illustrates the model registration method. Specifically, the computer acquires an initial ultrasound image and an initial image of the target cross-section. The initial ultrasound image is reconstructed in 3D to obtain an ultrasound 3D model of the target object; the initial image is also reconstructed in 3D to obtain an image 3D model of the target object, which includes the lesion region. Bounding box registration is performed based on the ultrasound 3D model and the image 3D model to obtain an initial linear transformation matrix. The initial linear transformation matrix is ​​used to perform coordinate transformation on the image 3D model to obtain a first transformed model. Iterative nearest neighbor registration is performed based on the ultrasound 3D model and the first transformed model to obtain a linear transformation matrix. The linear transformation matrix is ​​used to perform coordinate transformation on the first transformed model to obtain a second transformed model. Based on the ultrasound 3D model, a thin-plate spline algorithm is used to perform elastic registration on the second transformed model to obtain a nonlinear transformation matrix. The initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix are used to perform coordinate transformation on the lesion region of the model to obtain the transformed lesion region. The transformed lesion area is fused with the ultrasound 3D model to obtain the fused ultrasound model. The ultrasound image corresponding to the target cross section is determined from the fused ultrasound model and used as the stored target image.

[0143] The second method is an image registration method. Specifically, the computer acquires an initial ultrasound image and an initial image of the target cross-section, the initial image including the lesion region. Boundary box registration is performed based on the initial ultrasound image and the initial image to obtain an initial linear transformation matrix. The initial image is then subjected to coordinate transformation using the initial linear transformation matrix to obtain a first transformed image. Iterative nearest neighbor registration is performed based on the initial ultrasound image and the first transformed image to obtain a linear transformation matrix. The first transformed image is then subjected to coordinate transformation using the linear transformation matrix to obtain a second transformed image. Based on the initial ultrasound image, a thin-plate spline algorithm is used to perform elastic registration on the second transformed image to obtain a nonlinear transformation matrix. The lesion region is then subjected to coordinate transformation based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix to obtain the transformed lesion region. The transformed lesion region is then fused with the initial ultrasound image to obtain the stored target image.

[0144] like Figure 12 The diagram illustrates a method for registering real-time ultrasound images and stored target images. The computer equipment employs a feature point detection algorithm to extract a first set of feature points from the real-time ultrasound image and a second set of feature points from the stored target image. The first set of feature points includes at least one first feature point, and the second set of feature points includes at least one second feature point. The types of first and second feature points include corner points or points with significant gray-level gradient changes. For example... Figure 13 The image shown is a schematic diagram of corner points. Optionally, the feature point detection algorithm can be the Moravec feature point detection algorithm or the Harris corner feature point detection algorithm. Figure 14 The image shown is a schematic diagram of feature points in a real-time ultrasound image.

[0145] The computer device determines a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set, based on a first feature point set and a second feature point set. For example... Figure 15 The diagram illustrates the process of determining target feature point pairs. Specifically, for each first feature point in the first feature point set, a first target region containing the current first feature point is determined in the real-time ultrasound image, and a predetermined number of first pixels are acquired within the first target region. For each second feature point in the second feature point set, a second target region containing the current second feature point is determined in the existing target image, and a predetermined number of second pixels are acquired within the second target region. Figure 16 The diagram shows the determination of the first target region and the second target region.

[0146] For each first target region, based on the pixel value of the first pixel in the current first target region and the pixel value of the second pixel in each second target region, the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region is determined; if the similarity between the first target feature point and any second target feature point is greater than a preset value, then the first target feature point and the corresponding second target feature point are determined as a target feature point pair. For example... Figure 17 The diagram shown illustrates the determination of target feature point pairs.

[0147] The computer device determines the transformation function based on the transformation matrix and the coordinates of the feature points in the target feature point pair. Specifically, for each target feature point pair, based on the coordinates of the current target feature point pair, it determines whether the current target feature point pair is within the lesion region of the image; if it is, the weight corresponding to the current target feature point pair is reset to a first preset value; if it is not, the weight corresponding to the current target feature point pair is reset to a second preset value, wherein the first preset value is greater than the second preset value. Figure 18 The diagram illustrates the assignment of different weights to the current target feature point pairs inside and outside the lesion region of the image.

[0148] Based on the weights and transformation matrices of each target feature point pair, the objective function corresponding to each target feature point pair is determined. The transformation function is obtained by summing the objective functions corresponding to each target feature point pair.

[0149] Computer equipment updates the transformation matrix to change the function value of the transformation function accordingly. This process continues until the function value converges. The transformation matrix corresponding to this convergence is defined as the rigid transformation matrix. For example... Figure 19 The diagram illustrates how a rigid transformation matrix is ​​used to transform the coordinates of the lesion region in an image to obtain the target lesion region. The target lesion region is then fused with the real-time ultrasound image to obtain the registered ultrasound image.

[0150] In addition, based on the location of the lesion area indicated in the registered ultrasound image, the location of the lesion area is transformed into the robotic arm coordinate system using the conversion relationship between the image coordinate system and the robotic arm coordinate system. The robotic arm coordinate system is then used to guide the puncture needle for puncture. Figure 20 The diagram shown illustrates the change in puncture position.

[0151] The aforementioned image registration method acquires a real-time ultrasound image of the target object and a stored target image. It employs a feature point detection algorithm to extract a first set of feature points from the real-time ultrasound image and a second set of feature points from the stored target image, improving the accuracy of the extracted feature points and thus enhancing the precision of the image registration result. The target feature point pair consists of feature points from the first and second feature point sets. Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, a transformation function is determined. The transformation matrix is ​​updated to change the function value accordingly. This process continues until the function value converges, and the transformation matrix corresponding to this convergence is defined as the rigid transformation matrix. The coordinates of the image lesion region are transformed using this rigid transformation matrix to obtain the target lesion region. Since the stored target image includes the image lesion region, this method of using the transformation matrix corresponding to the convergence of the transformation function to transform the coordinates of the image lesion region ensures that the obtained target lesion region is closer to the actual lesion region. Fusing the target lesion region with the real-time ultrasound image results in a registered ultrasound image with high accuracy, thus improving the precision of the registration result.

[0152] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0153] Based on the same inventive concept, this application also provides an image registration apparatus for implementing the image registration method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more image registration apparatus embodiments provided below can be found in the limitations of the image registration method described above, and will not be repeated here.

[0154] In one embodiment, such as Figure 21 As shown, an image registration device 100 is provided, including: an image acquisition module 110, a feature point extraction module 120, a feature point pair determination module 130, a rigid transformation matrix determination module 140, and a fusion module 150, wherein:

[0155] Image acquisition module 110 is used to acquire real-time ultrasound images of the target object and stored target images, the stored target images including the lesion area in the image;

[0156] The feature point extraction module 120 is used to extract the first set of feature points in the real-time ultrasound image and the second set of feature points in the stored target image using a feature point detection algorithm.

[0157] The feature point pair determination module 130 is used to determine a target feature point pair composed of feature points in the first feature point set and feature points in the second feature point set based on the first feature point set and the second feature point set.

[0158] The rigid transformation matrix determination module 140 is used to determine the transformation function based on the transformation matrix and the coordinates of the feature points in the target feature point pair. The transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix corresponding to the convergence of the function value is determined as the rigid transformation matrix.

[0159] The fusion module 150 is used to transform the coordinates of the image lesion region based on the rigid transformation matrix to obtain the target lesion region; and to fuse the target lesion region with the real-time ultrasound image to obtain the registered ultrasound image.

[0160] The aforementioned image registration device acquires real-time ultrasound images of the target object and stored target images. It employs a feature point detection algorithm to extract a first set of feature points from the real-time ultrasound image and a second set of feature points from the stored target image, improving the accuracy of the extracted feature points and thus enhancing the precision of the image registration results. The target feature point pair consists of feature points from the first and second feature point sets. Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, a transformation function is determined. The transformation matrix is ​​updated to change the function value accordingly. This process continues until the function value converges, and the transformation matrix corresponding to this convergence is defined as the rigid transformation matrix. The coordinates of the image lesion region are transformed using this rigid transformation matrix to obtain the target lesion region. Since the stored target image includes the image lesion region, this method of using the transformation matrix corresponding to the convergence of the transformation function to transform the coordinates of the image lesion region ensures that the obtained target lesion region is closer to the actual lesion region. Fusing the target lesion region with the real-time ultrasound image results in a registered ultrasound image with high accuracy, improving the precision of the registration results.

[0161] In one embodiment, the first feature point set includes at least one first feature point, and the second feature point set includes at least one second feature point. Regarding determining a target feature point pair composed of feature points from the first and second feature point sets, the feature point pair determination module 130 is further configured to: for each first feature point in the first feature point set, determine a first target region in the real-time ultrasound image containing the current first feature point, and acquire a preset number of first pixels in the first target region; for each second feature point in the second feature point set, determine a second target region in an existing target image containing the current second feature point, and acquire a preset number of second pixels in the second target region; for each first target region, based on the pixel value of the first pixel in the current first target region and the pixel value of the second pixel in each second target region, determine the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region; if the similarity between the first target feature point and any second target feature point is greater than a preset value, then determine the first target feature point and the corresponding second target feature point as a target feature point pair.

[0162] In one embodiment, in determining the similarity between a first target feature point corresponding to the current first target region and a second target feature point corresponding to each second target region based on the pixel value of a first pixel point in the current first target region and the pixel value of a second pixel point in each second target region, the feature point pair determination module 130 is further configured to: for each second target region, obtain a first difference between the pixel value of each first pixel point in the current first target region and the pixel value of each second pixel point in the current second target region; obtain a second difference between the first pixel average value of the current first target region and the second pixel average value of the current second target region; obtain a histogram similarity between the first pixel histogram of the current first target region and the second pixel histogram of the current second target region; use the sum of squares of each first difference as a first pixel difference; use the sum of squares of each second difference as a second pixel difference; use the histogram similarity as a third pixel difference; and determine the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region based on the first pixel difference, the second pixel difference, and the third pixel difference.

[0163] In one embodiment, in determining the transformation function based on the transformation matrix and the coordinates of the feature points in the target feature point pair, the rigid transformation matrix determination module 140 is further configured to: for each target feature point pair in the target feature point pair, determine whether the current target feature point pair is within the image lesion region based on the coordinates of the current target feature point pair; if it is, reset the weight corresponding to the current target feature point pair to a first preset value; if it is not, reset the weight corresponding to the current target feature point pair to a second preset value; wherein the first preset value is greater than the second preset value; determine the objective function corresponding to each target feature point pair based on the weights corresponding to each target feature point pair and the transformation matrix; sum the objective functions corresponding to each target feature point pair to obtain the transformation function.

[0164] In one embodiment, regarding the acquisition of a stored target image, the image acquisition module 110 is further configured to: acquire an initial ultrasound image of the target object and an initial image of the target object; perform three-dimensional reconstruction on the initial ultrasound image to obtain an ultrasound three-dimensional model of the target object; perform three-dimensional reconstruction on the initial image to obtain an image three-dimensional model of the target object; and register the ultrasound three-dimensional model and the image three-dimensional model to obtain a stored target image.

[0165] In one embodiment, the image acquisition module 110 further comprises: performing bounding box registration based on the ultrasound three-dimensional model and the image three-dimensional model to obtain an initial linear transformation matrix; performing coordinate transformation on the image three-dimensional model using the initial linear transformation matrix to obtain a first transformation model; performing iterative nearest neighbor registration based on the ultrasound three-dimensional model and the first transformation model to obtain a linear transformation matrix; performing coordinate transformation on the first transformation model using the linear transformation matrix to obtain a second transformation model; performing elastic registration on the second transformation model based on the ultrasound three-dimensional model using a thin plate spline algorithm to obtain a nonlinear transformation matrix; and obtaining the stored target image based on the initial linear transformation matrix, the linear transformation matrix, the nonlinear transformation matrix, the model lesion region, and the ultrasound three-dimensional model.

[0166] In one embodiment, in obtaining a stored target image based on an initial linear transformation matrix, a linear transformation matrix, a nonlinear transformation matrix, a lesion region, and an ultrasound three-dimensional model, the image acquisition module 110 is further configured to: perform coordinate transformation on the model lesion region based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix to obtain a transformed lesion region; fuse the transformed lesion region with the ultrasound three-dimensional model to obtain a fused ultrasound model; determine the ultrasound image corresponding to the target object from the fused ultrasound model, and use the ultrasound image as a stored target image.

[0167] In one embodiment, regarding the acquisition of the existing target image, the image acquisition module 110 is further configured to: acquire an initial ultrasound image of the target object and an initial image of the target object; the initial image includes an image lesion region; perform bounding box registration based on the initial ultrasound image and the initial image to obtain an initial linear transformation matrix; perform coordinate transformation on the initial image using the initial linear transformation matrix to obtain a first transformed image; perform iterative nearest neighbor registration based on the initial ultrasound image and the first transformed image to obtain a linear transformation matrix; perform coordinate transformation on the first transformed image using the linear transformation matrix to obtain a second transformed image; perform elastic registration on the second transformed image using a thin-plate spline algorithm based on the initial ultrasound image to obtain a nonlinear transformation matrix; perform coordinate transformation on the image lesion region based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix to obtain a transformed lesion region; and fuse the transformed lesion region with the initial ultrasound image to obtain the existing target image.

[0168] Each module in the aforementioned image registration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0169] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 22 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image registration method.

[0170] Those skilled in the art will understand that Figure 22The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0172] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0174] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image registration method, characterized in that, The method includes: Acquire real-time ultrasound images of the target object and stored target images, wherein the stored target images include the lesion region in the image; A feature point detection algorithm is used to extract the first set of feature points from the real-time ultrasound image and the second set of feature points from the stored target image, respectively. Based on the first feature point set and the second feature point set, a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set is determined. Based on the transformation matrix and the coordinates of the feature points in the target feature point pair, a transformation function is determined. The transformation matrix is ​​updated to change the function value accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix corresponding to the convergence of the function value is determined as the rigid transformation matrix. The coordinates of the image lesion region are transformed based on the rigid transformation matrix to obtain the target lesion region. The target lesion region is fused with the real-time ultrasound image to obtain the registered ultrasound image. The steps for acquiring the stored target image include: Acquire the initial ultrasound image and the initial image of the target object; The initial ultrasound image is reconstructed in three dimensions to obtain an ultrasound three-dimensional model of the target object; the initial image is reconstructed in three dimensions to obtain an image three-dimensional model of the target object. The ultrasound 3D model and the image 3D model are registered to obtain the stored target image; The three-dimensional image model of the target object includes the lesion area of ​​the model; The process of registering the ultrasound 3D model and the image 3D model to obtain the stored target image includes: Based on the ultrasound 3D model and the image 3D model, bounding box registration is performed to obtain the initial linear transformation matrix; The initial linear transformation matrix is ​​used to perform coordinate transformation on the image 3D model to obtain a first transformation model; based on the ultrasound 3D model and the first transformation model, iterative nearest neighbor registration is performed to obtain a linear transformation matrix; The first transformation model is subjected to coordinate transformation using the linear transformation matrix to obtain the second transformation model; based on the ultrasonic three-dimensional model, the second transformation model is elastically registered using the thin plate spline algorithm to obtain the nonlinear transformation matrix; Based on the initial linear transformation matrix, the linear transformation matrix, the nonlinear transformation matrix, the model lesion region, and the ultrasound three-dimensional model, the stored target image is obtained.

2. The method according to claim 1, characterized in that, The first set of feature points includes at least one first feature point, and the second set of feature points includes at least one second feature point; The step of determining a target feature point pair composed of feature points from the first feature point set and feature points from the second feature point set based on the first feature point set and the second feature point set includes: For each first feature point in the first feature point set, determine a first target region in the real-time ultrasound image that contains the current first feature point, and obtain a preset number of first pixels in the first target region; For each second feature point in the second feature point set, determine a second target region in the stored target image that contains the current second feature point, and obtain the preset number of second pixels in the second target region; For each first target region, based on the pixel value of the first pixel in the current first target region and the pixel value of the second pixel in each second target region, the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region is determined; if the similarity between the first target feature point and any second target feature point is greater than a preset value, then the first target feature point and the corresponding second target feature point are determined as a target feature point pair.

3. The method according to claim 2, characterized in that, The step of determining the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region based on the pixel value of the first pixel point in the current first target region and the pixel value of the second pixel point in each second target region includes: For each second target region, obtain the first difference between the pixel value of each first pixel in the current first target region and the pixel value of each second pixel in the current second target region; obtain the first pixel average value of the current first target region and the second pixel average value of the current second target region; obtain the histogram similarity between the first pixel histogram of the current first target region and the second pixel histogram of the current second target region. The sum of the squares of each first difference is taken as the first pixel difference; The sum of the squares of each second difference is taken as the second pixel difference; The histogram similarity is used as the third pixel difference; Based on the first pixel difference, the second pixel difference, and the third pixel difference, the similarity between the first target feature point corresponding to the current first target region and the second target feature point corresponding to each second target region is determined.

4. The method according to claim 1, characterized in that, The step of determining the transformation function based on the transformation matrix and the coordinates of the feature points in the target feature point pair includes: For each target feature point pair in the target feature point pair, based on the coordinates of the current target feature point pair, determine whether the current target feature point pair is within the lesion area of ​​the image; if it is, reset the weight corresponding to the current target feature point pair to a first preset value; if it is not, reset the weight corresponding to the current target feature point pair to a second preset value; wherein, the first preset value is greater than the second preset value; Based on the weights and transformation matrices of each target feature point pair, the objective function corresponding to each target feature point pair is determined. The transformation function is obtained by summing the values ​​of each target feature point with respect to the corresponding objective function.

5. The method according to claim 1, characterized in that, The process of obtaining the stored target image based on the initial linear transformation matrix, the linear transformation matrix, the nonlinear transformation matrix, the model lesion region, and the ultrasound three-dimensional model includes: Based on the initial linear transformation matrix, the linear transformation matrix, and the nonlinear transformation matrix, the coordinate transformation of the model lesion region is performed to obtain the transformed lesion region; The transformed lesion area is fused with the ultrasound 3D model to obtain the fused ultrasound model; The ultrasound image corresponding to the target object is determined from the fused ultrasound model, and the ultrasound image is used as the stored target image.

6. The method according to claim 1, characterized in that, The acquisition of real-time ultrasound images and stored target images of the target object includes: Acquire real-time ultrasound images of the target cross-section and existing target images of the target object; the cross-section is a section perpendicular to the axis of the target object.

7. The method according to claim 1, characterized in that, The step of fusing the target lesion region with the real-time ultrasound image to obtain a registered ultrasound image includes: Based on the coordinates of the target lesion area, the target lesion area is delineated in the real-time ultrasound image to obtain the delineated image, which is then used as the registered ultrasound image.

8. An image registration device, characterized in that, The device includes: The image acquisition module is used to acquire real-time ultrasound images of the target object and stored target images, wherein the stored target images include the image lesion region; The feature point extraction module is used to extract the first set of feature points in the real-time ultrasound image and the second set of feature points in the stored target image using a feature point detection algorithm. The feature point pair determination module is used to determine a target feature point pair composed of feature points in the first feature point set and feature points in the second feature point set based on the first feature point set and the second feature point set; The rigid transformation matrix determination module is used to determine the transformation function based on the transformation matrix and the coordinates of the feature points in the target feature point pair. The transformation matrix is ​​updated so that the function value of the transformation function changes accordingly. The transformation matrix is ​​continuously updated until the function value converges. The transformation matrix corresponding to the convergence of the function value is determined as the rigid transformation matrix. The fusion module is used to transform the coordinates of the image lesion region based on the rigid transformation matrix to obtain the target lesion region; and to fuse the target lesion region with the real-time ultrasound image to obtain the registered ultrasound image. The image acquisition module is further configured to: acquire an initial ultrasound image of the target object and an initial image of the target object; perform three-dimensional reconstruction on the initial ultrasound image to obtain an ultrasound three-dimensional model of the target object; perform three-dimensional reconstruction on the initial image to obtain an image three-dimensional model of the target object; and register the ultrasound three-dimensional model and the image three-dimensional model to obtain a stored target image. The three-dimensional image model of the target object includes a lesion region. The image acquisition module is further configured to: perform bounding box registration based on the ultrasound three-dimensional model and the image three-dimensional model to obtain an initial linear transformation matrix; perform coordinate transformation on the image three-dimensional model using the initial linear transformation matrix to obtain a first transformation model; perform iterative nearest neighbor registration based on the ultrasound three-dimensional model and the first transformation model to obtain a linear transformation matrix; perform coordinate transformation on the first transformation model using the linear transformation matrix to obtain a second transformation model; perform elastic registration on the second transformation model using a thin-plate spline algorithm based on the ultrasound three-dimensional model to obtain a nonlinear transformation matrix; and obtain a stored target image based on the initial linear transformation matrix, the linear transformation matrix, the nonlinear transformation matrix, the lesion region, and the ultrasound three-dimensional model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images