Image registration method, device, apparatus and storage medium

Abstract

This application discloses an image registration method, apparatus, device, and storage medium, relating to the field of artificial intelligence. The method includes: acquiring a pair of images to be registered, the pair comprising a first image and a second image; performing edge extraction on the first and second images using an edge extraction network to obtain a first contour map and a second contour map; in the k-th iteration, performing contour alignment on the first and second contour maps based on the k-th transformation matrix, and updating the k-th transformation matrix to the (k+1)-th transformation matrix based on the contour alignment result of the k-th iteration; and, if the iteration termination condition is met, determining the transformation matrix obtained in the last iteration as the image transformation matrix, which is used for image registration of the first and second images. The method provided in this application helps improve the accuracy of image registration.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image registration method, apparatus, device, and storage medium. Background Technology

[0002] Image registration is the process of performing a projection transformation on an image to align it with a reference image in the same image coordinate system.

[0003] Image registration is the process of finding the transformation matrix between images. In related techniques, the transformation matrix is ​​calculated based on the feature points of the images. In this process, key feature points in the images to be matched are first extracted, then these key feature points are matched, and the transformation matrix is ​​calculated based on the matched feature point pairs.

[0004] The above method relies on the extraction of accurate feature points. If good feature points cannot be extracted, the transformation matrix between images cannot be accurately estimated, which affects the accuracy of image matching. Summary of the Invention

[0005] This application provides an image registration method, apparatus, device, and storage medium, which can improve the accuracy of image registration. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide an image registration method, the method comprising:

[0007] Obtain the registration image pair to be registered, wherein the registration image pair includes a first image and a second image;

[0008] Edge extraction is performed on the first image and the second image using an edge extraction network to obtain a first contour map and a second contour map;

[0009] In the k-th iteration process, the first contour map and the second contour map are aligned in the k-th round based on the k-th transformation matrix, and the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th contour alignment result. The contour alignment difference corresponding to the updated (k+1)-th transformation matrix is ​​less than the contour alignment difference corresponding to the k-th transformation matrix, where k is a positive integer.

[0010] If the iteration termination condition is met, the transformation matrix obtained in the last round is determined as the image transformation matrix, which is used to perform image registration between the first image and the second image.

[0011] On the other hand, embodiments of this application provide an image registration apparatus, the apparatus comprising:

[0012] The image acquisition module is used to acquire a pair of images to be registered, wherein the pair of images includes a first image and a second image;

[0013] An edge extraction module is used to extract edges from the first image and the second image through an edge extraction network to obtain a first contour map and a second contour map.

[0014] The matrix update module is used to perform contour alignment of the first contour map and the second contour map based on the kth transformation matrix during the kth iteration, and to update the kth transformation matrix to the (k+1)th transformation matrix based on the contour alignment result of the kth iteration, wherein the contour alignment difference corresponding to the updated (k+1)th transformation matrix is ​​less than the contour alignment difference corresponding to the kth transformation matrix, and k is a positive integer;

[0015] The matrix determination module is used to determine the transformation matrix obtained in the last round as the image transformation matrix when the iteration termination condition is met. The image transformation matrix is ​​used to perform image registration between the first image and the second image.

[0016] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the image registration method as described above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image registration method as described above.

[0018] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image registration method provided in the above aspects.

[0019] The beneficial effects of the technical solutions provided in this application include at least the following:

[0020] In this embodiment, for the first image and the second image to be registered, the computer device can first perform edge extraction on the images to obtain the first contour map and the second contour map corresponding to the first image, and then perform contour alignment based on the contour maps to obtain the transformation matrix between the first image and the second image. The transformation matrix is ​​determined by contour alignment without relying on the extraction of feature points. Furthermore, by iteratively aligning the contour maps between the first and second contour maps, the image transformation matrix is ​​obtained with the goal of minimizing the contour alignment difference, which can improve the accuracy of estimating the image transformation matrix and help improve the accuracy of image registration. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown;

[0023] Figure 2 A flowchart illustrating an exemplary embodiment of the image registration method provided in this application is shown;

[0024] Figure 3 A flowchart of an image registration method provided in another exemplary embodiment of this application is shown;

[0025] Figure 4 This invention provides a schematic diagram of the contour point alignment method according to an exemplary embodiment of the present application.

[0026] Figure 5 A flowchart illustrating the determination of an image transformation matrix is ​​shown in an exemplary embodiment of this application;

[0027] Figure 6 A flowchart of an image registration method provided in another exemplary embodiment of this application is shown;

[0028] Figure 7 This invention provides a schematic diagram of the structure of an image alignment method according to an exemplary embodiment of the present application.

[0029] Figure 8 A flowchart illustrating a training method for an edge extraction network provided in an exemplary embodiment of this application is shown;

[0030] Figure 9 This is a structural block diagram of an image registration apparatus provided in an exemplary embodiment of this application;

[0031] Figure 10 A schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0033] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0034] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0035] In related technologies, the registration process using the feature point method relies on the extraction of feature points. Therefore, in this application embodiment, an image registration method is provided, which performs edge extraction on image pairs and then performs contour alignment based on the contour map obtained from the edge extraction. The image transformation matrix is ​​determined based on the contour alignment process, which can improve the accuracy of image registration.

[0036] The image registration method provided in this application can be applied to industrial registration scenarios. In one possible implementation, it can be applied to industrial quality inspection scenarios, where quality inspection images are typically captured under different conditions. Computer equipment can use the image registration method provided in this application to register the quality inspection images captured under different conditions with a reference image, thereby detecting defects.

[0037] In addition to the above-mentioned application scenarios, the image registration method provided in this application embodiment can also be applied to other image registration scenarios. This application embodiment does not limit the specific application scenarios.

[0038] Figure 1A schematic diagram of an implementation environment provided by an exemplary embodiment of this application is shown. This implementation environment includes a computer device 110 and a server 120. The computer device 110 and the server 120 communicate via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0039] Computer device 110 is an electronic device that requires image registration. This electronic device can be a smartphone, tablet, or personal computer, etc., and this embodiment is not limited thereto. In some embodiments, computer device 110 runs an application with image registration functionality. When it is necessary to register a pair of images, the user inputs the image pair into the application, thereby uploading the image pair to server 120. Server 120 performs image registration and provides the registration result.

[0040] Server 120 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0041] In some embodiments, server 120 is used to provide image registration services for applications running on computer device 110. In one possible implementation, after receiving a pair of registered images sent by computer device 110, server 120 registers the pair of registered images using an image registration method and returns the registered image to computer device 110 so that computer device 110 can display the image registration result.

[0042] Of course, in other possible implementations, image registration can also be performed locally by computer device 110 without the aid of server 120, and this embodiment does not limit this. Furthermore, for ease of description, the following embodiments are illustrated using the example of image registration being performed by a computer device.

[0043] Optionally, computer equipment includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0044] Please refer to Figure 2This document illustrates a flowchart of an image registration method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example for illustration, and the method includes the following steps.

[0045] Step 201: Obtain the registration image pair to be registered, which includes a first image and a second image.

[0046] Here, a registered image pair refers to a pair of images of the same object acquired under different conditions. Optionally, these can be image pairs acquired at different times, from different angles, or through different acquisition devices.

[0047] The image registration pair includes a first image and a second image. In one possible implementation, the computer device can select one of the first image and the second image as a reference image, and then perform image transformation on the other image based on the reference image to achieve image registration between the first image and the second image. Illustratively, the first image can be the reference image, and the second image can be the image to be registered.

[0048] Step 202: Edge extraction is performed on the first image and the second image using an edge extraction network to obtain a first contour map and a second contour map.

[0049] During image registration, it is necessary to estimate the optimal image transformation matrix to minimize the differences between the images after alignment. In this embodiment, the process of determining the image transformation matrix includes a contour map extraction process and a contour alignment process. The image transformation matrix is ​​determined by minimizing the image alignment differences through the contour alignment process.

[0050] Therefore, in one possible implementation, the computer device first performs edge extraction on the first image and the second image through an edge extraction network to obtain a first contour map corresponding to the first image and a second contour map corresponding to the second image.

[0051] The edge extraction network is trained using a semi-supervised approach. During training, the network undergoes multi-stage training using pseudo-labels corresponding to sample images. These pseudo-labels are contour maps obtained by edge detection algorithms on the sample images. This training method allows the trained edge extraction network to retain accurate edge localization and improves the suppression of internal details, thus enhancing the accuracy of edge extraction. Optionally, the edge extraction network can be a pixel difference network (PiDiNet).

[0052] Step 203: During the k-th iteration, the first contour map and the second contour map are aligned based on the k-th transformation matrix, and the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th contour alignment result. The contour alignment difference corresponding to the updated (k+1)-th transformation matrix is ​​less than the contour alignment difference corresponding to the k-th transformation matrix, where k is a positive integer.

[0053] After obtaining the first and second contour images, the computer device can obtain the image transformation matrix through an iterative alignment process of the first and second contour images.

[0054] In each iteration, the transformation matrix obtained in the previous round is updated to obtain a new transformation matrix. In the k-th iteration, the computer device can use the k-th transformation matrix obtained in the previous iteration to perform k-th contour alignment on the first contour map and the second contour map, thereby updating the current k-th transformation matrix according to the contour alignment result of the k-th round, obtaining the (k+1)-th transformation matrix, and entering the k-th iteration process.

[0055] Furthermore, the iterative alignment process aims to minimize the alignment difference. In each iteration, the transformation matrix is ​​updated, and the corresponding contour alignment difference is reduced. That is, in the k-th iteration, the k-th transformation matrix is ​​updated to obtain the (k+1)-th transformation matrix. After aligning the first and second contour images using the (k+1)-th transformation matrix, the contour alignment difference is less than that after aligning the first and second contour images using the k-th transformation matrix.

[0056] Optionally, the contour alignment method may include at least one of contour point alignment and image alignment. In one possible implementation, the computer device may iteratively obtain the image transformation matrix through multiple rounds of contour point alignment; or, the computer device may iteratively obtain the image transformation matrix through multiple rounds of image alignment; or, the computer device may obtain a first image transformation matrix through multiple rounds of contour point alignment and a second image transformation matrix through multiple rounds of image alignment, and finally obtain the image transformation matrix based on the first image transformation matrix and the second image transformation matrix.

[0057] Step 204: If the iteration termination condition is met, the transformation matrix obtained in the last round is determined as the image transformation matrix. The image transformation matrix is ​​used to perform image registration between the first image and the second image.

[0058] In one possible implementation, an error function can be set for the alignment difference of the images. When the error function converges, it is determined that the iteration termination condition is met, that is, when the image alignment difference is minimized, the iteration termination condition is determined to be met. The transformation matrix obtained in the last round can be determined as the image transformation matrix.

[0059] Alternatively, in another possible implementation, when the number of iterations reaches the iteration threshold, the iteration process is terminated, and the transformation matrix obtained in the last round is determined as the image transformation matrix to avoid continuous iteration.

[0060] After obtaining the image transformation matrix, the computer device can use the image transformation matrix to transform the image to be registered, thereby achieving image registration between the image to be registered and the reference image.

[0061] In this embodiment, for the first image and the second image to be registered, the computer device can first perform edge extraction on the images to obtain the first contour map and the second contour map corresponding to the first image, and then perform contour alignment based on the contour maps to obtain the transformation matrix between the first image and the second image. The transformation matrix is ​​determined by contour alignment without relying on the extraction of feature points. Furthermore, by iteratively aligning the contour maps between the first and second contour maps, the image transformation matrix is ​​obtained with the goal of minimizing the contour alignment difference, which can improve the accuracy of estimating the image transformation matrix and help improve the accuracy of image registration.

[0062] In one possible implementation, contour points can be extracted from the first and second contour maps to obtain contour points, and then the image transformation matrix can be obtained using a contour point alignment process. An exemplary embodiment will be described below.

[0063] Please refer to Figure 3 This illustration shows a flowchart of an image registration method provided in another exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example for explanation, and the method includes the following steps.

[0064] Step 301: Obtain the registration image pair to be registered.

[0065] Step 302: Edge extraction is performed on the first image and the second image using an edge extraction network to obtain a first contour map and a second contour map.

[0066] The implementation methods of steps 301 and 302 can refer to steps 201 and 202 above, and will not be repeated in this embodiment.

[0067] Step 303: Filter the first contour point set from the first contour map and filter the second contour point set from the second contour map.

[0068] In one possible implementation, the computer device first converts the first contour image and the second contour image into 2D contour points, and then aligns them based on the two contour point sets. The process of converting the contour image into 2D contour points is equivalent to filtering pixels within the contour image. In this embodiment, the contour image extracted by the edge extraction network contains contour values ​​corresponding to each pixel. These contour values ​​indicate the probability that the pixel lies within the image contour. The contour values ​​range from 0 to 1, and the contour values ​​are positively correlated with the probability.

[0069] In the process of converting a contour map into 2D contour points, the computer device can filter based on the contour value corresponding to each pixel. If the contour value is greater than the contour threshold, the corresponding pixel can be designated as a contour point. For example, the contour threshold can be 0.8.

[0070] By filtering through a contour threshold, the set of contour points corresponding to the contour map can be obtained. However, the contour point set may be dense and may contain redundant contour points. Therefore, in one possible implementation, the computer device can further filter the candidate contour point set obtained through the contour threshold using a Non-Maximum Suppression (NMS) algorithm. During the filtering process, the contour point with the highest contour value can be selected from among the n locally adjacent candidate contour points. The contour point with the highest contour value is retained, and other contour points are deleted, thereby obtaining a sparse contour point set. For example, the contour point with the highest contour value can be selected and retained from among the 20 locally adjacent candidate contour points.

[0071] Furthermore, to further improve the efficiency of contour point selection, the contour map can be evenly divided into several contour sub-regions, and then the NMS algorithm can be used to select several contour sub-regions to obtain the selected contour point set.

[0072] The computer device can filter the first contour map to obtain the first contour point set and filter the second contour map to obtain the second contour point set, thereby aligning the first contour point set with the second contour point set.

[0073] Indicative, such as Figure 4 As shown, edge extraction is performed on the reference image (first image) 401 and the image to be registered (second image) 402 to obtain a first contour image 403 and a second contour image 404. Then, the first contour image 403 is transformed into 2D contour points to obtain a first contour point set 405, and the second contour image 404 is transformed into 2D contour points to obtain a second contour point set 406. The first image and the second image are part images obtained under different conditions from photographs of the same part.

[0074] Step 304: During the k-th iteration, the contour points in the first contour map and the second contour map are aligned in the k-th round based on the k-th transformation matrix, and the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th round point set alignment result.

[0075] In one possible implementation, the computer device determines the image transformation matrix by performing a multi-round iterative alignment process on a first contour point set and a second contour point set, based on the correspondence between matching points in the first and second contour point sets. Optionally, the Iterative Closest Point (ICP) algorithm can be used to iteratively obtain the image transformation matrix. The iterative process will be illustrated below.

[0076] In the k-th iteration process, the process of aligning the point set in the k-th iteration and obtaining the (k+1)-th transformation matrix based on the alignment result of the k-th iteration can include steps 304a-304c (not shown in the figure):

[0077] Step 304a: Perform the kth round of point set transformation on the second contour point set based on the kth transformation matrix to obtain the kth contour point set.

[0078] During the alignment of the first contour point set and the second contour point set, a transformation matrix can be used to transform the contour point set corresponding to the image to be registered, thereby aligning it with the contour point set corresponding to the reference image. In this embodiment, the reference image is used as the first image, and the image to be registered is used as the second image for explanation. That is, during the alignment of the first contour point set and the second contour point set, a transformation matrix can be used to transform the second contour point set, aligning the transformed second contour point set with the first contour point set.

[0079] In the k-th iteration, the computer device first transforms the second contour point set using the k-th transformation matrix to obtain the k-th contour point set after the k-th transformation.

[0080] Step 304b: Based on the distance between each point in the first set of contour points and the set of k contour points, determine the k-th round of matching contour point pairs.

[0081] In determining the transformation matrix using the alignment process of contour point pairs, the transformation matrix can be obtained based on the coordinates of different point pairs. Therefore, in one possible implementation, during each iteration, the computer device determines matching contour point pairs between the first contour point set and the second contour point set.

[0082] Since the matching relationship between the first contour point set and the second contour point set cannot be directly determined, the computer device can determine the matching contour point pair based on the distance between each point in the transformed second contour point set and the first contour point set.

[0083] In the k-th iteration, the computer device can determine the k-th contour point in the k-th contour point set that is closest to the contour points in the first contour point set based on the distances between the k-th contour point set and the points in the first contour point set. Therefore, the second contour point corresponding to the k-th contour point can be determined as the contour point that matches the first contour point. The method for determining the matching contour point pair may include the following steps:

[0084] Step 1: Using the nearest neighbor search method, find the kth contour point that matches the first contour point in the kth contour point set. The first contour point belongs to the first contour point set.

[0085] The nearest neighbor search method refers to finding the point with the smallest distance to the query point in a given set. In one possible implementation, the computer device uses the nearest neighbor search method to find the k-th contour point with the smallest distance to the first contour point in the k-th contour point set, thereby determining the k-th contour point as the contour point in the k-th contour point set that matches the first contour point.

[0086] Step 2: If the distance between the first contour point and the kth contour point is less than the distance error threshold, determine the second contour point corresponding to the first contour point and the kth contour point as a matching contour point pair.

[0087] After finding the kth contour point corresponding to each first contour point, there may be a problem that the distance between the first contour point and the kth contour point is too far, and they may not actually be a matching pair. Therefore, the computer device needs to filter them by a distance error threshold. When the distance between the first contour point and the kth contour point is less than the distance error threshold, the second contour point corresponding to the first contour point and the kth contour point is determined as a matching contour point pair.

[0088] For illustration, the distance error threshold can be 1mm. The point with the smallest distance between each point in the k-th contour point set and the first contour point A in the first contour point set is determined by the nearest neighbor search method. The point is the k-th contour point B. If the distance between the first contour point A and the k-th contour point B is less than 1mm, then the second contour point corresponding to the first contour point A and the k-th contour point B can be determined as a matching contour point pair.

[0089] Using the above method, the first and second contour points matched in the k-th iteration can be determined, thus obtaining the matching point pair in the k-th iteration.

[0090] Step 304c: Based on the correspondence between the contour point pairs matched in the kth round, determine the (k+1)th transformation matrix.

[0091] Once the k-th round of matching contour point pairs is determined, the transformation matrix can be obtained based on the coordinates between each matching point pair. Since there may be multiple matching contour point pairs, but only a few matching pairs may be needed when actually calculating the transformation matrix, in one possible implementation, the (k+1)-th transformation matrix is ​​estimated using the Random Sample Consensus (RANSAC) algorithm.

[0092] The process of estimating the transformation matrix is ​​also an iterative calculation. In the t-th iteration, the computer randomly samples the minimum number of matching pairs from the t-th matching point set. Based on these sampled matching pairs, the t-th transformation matrix is ​​calculated. The minimum number of matching pairs can be determined based on the minimum number of point pairs required to calculate the transformation matrix. For example, if calculating the transformation matrix requires at least 4 matching pairs, then 4 matching pairs can be sampled.

[0093] Next, the computer equipment uses the t-th transformation matrix to transform the second contour point in the matching pairs (excluding the sample matching pairs) in the t-th matching point set. If the distance error between the transformed second contour point and the first contour point is less than the distance error threshold, the matching point pair corresponding to the first and second contour points can be added to the t-th matching point set. If the number of matching point pairs in the t-th matching point set is greater than the number of matching point pairs in the (t-1)-th matching point set, the (t+1)-th matching point set is determined, and the (t+1)-th iteration process begins. Finally, if the iteration count threshold is reached, the transformation matrix output in the last round is determined as the (k+1)-th transformation matrix. Alternatively, if the number of matching point pairs in the t-th matching point set reaches the proportion threshold, the t-th transformation matrix is ​​determined as the (k+1)-th transformation matrix. For illustration, the proportion threshold can be 95%.

[0094] Step 305: If the first error function converges, the transformation matrix obtained in the last round is determined as the image transformation matrix. The first error function is used to represent the distance error between the matching contour point pairs in the first contour point set and the transformed second contour point set.

[0095] In one possible implementation, the computer device measures the difference in the transformation matrix used for contour point alignment using a first error function. The first error function is shown below:

[0096]

[0097] in, For the first contour point set, This represents the set of second contour points after transformation by the transformation matrix H.

[0098] When the latest transformation matrix is ​​obtained in each iteration, the value of the first error function corresponding to the first error function can be recalculated. When the first error function converges, the computer device can stop the iteration and determine the transformation matrix obtained in the last round as the image transformation matrix.

[0099] Step 306: If the iteration count threshold is reached, the transformation matrix obtained in the last round is determined as the image transformation matrix.

[0100] Alternatively, in another possible implementation, to avoid continuous iteration, when the number of iterations reaches an iteration threshold, the transformation matrix obtained in the last round is determined as the image transformation matrix. For example, the iteration threshold could be 100.

[0101] Indicative, such as Figure 4 As shown, after obtaining the first contour point set 405 and the second contour point set 406, the image transformation matrix H is obtained through the ICP algorithm iteration.

[0102] The process of iteratively determining the image transformation matrix can be as follows: Figure 5 As shown, it includes the following steps:

[0103] Step 501: During the k-th iteration, the second contour point set is transformed using the k-th transformation matrix.

[0104] Step 502: Establish a matching relationship between the first contour point and the second contour point based on the first contour point set and the transformed second contour point set using the nearest neighbor search method.

[0105] Step 503: Remove matching point pairs whose error exceeds the threshold.

[0106] Step 504: Estimate the (k+1)th transformation matrix using the RANSAC algorithm.

[0107] Step 505: Determine whether the first error function has converged or whether the iteration number threshold has been reached. If yes, proceed to step 506; otherwise, proceed to step 501.

[0108] Step 506: Output the image transformation matrix.

[0109] In this embodiment, the first contour image and the second contour image are aligned by contour point alignment. The transformation matrix between the first contour image and the second contour image is obtained iteratively through multiple rounds of contour point alignment, which can improve the accuracy of the image transformation matrix and thus improve the accuracy of image registration.

[0110] In the above embodiments, contour alignment is performed using contour point alignment. In another possible implementation, the first contour image and the second contour image can be aligned directly using image alignment, thereby obtaining the image transformation matrix through multiple rounds of image alignment iteration. Exemplary embodiments will be described below.

[0111] Please refer to Figure 6 This illustration shows a flowchart of an image registration method provided in another exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example for explanation, and the method includes the following steps.

[0112] Step 601: Obtain the registration image pair to be registered.

[0113] Step 602: Edge extraction is performed on the first image and the second image using an edge extraction network to obtain a first contour map and a second contour map.

[0114] The implementation methods of steps 601 and 602 can refer to steps 201 and 202 above, and will not be repeated in this embodiment.

[0115] Step 603: Perform image alignment of the first contour map and the second contour map in the kth round based on the kth transformation matrix, and update the kth transformation matrix to the (k+1)th transformation matrix based on the kth round image alignment result.

[0116] In one possible implementation, the computer device can directly align the first contour image and the second contour image. During each alignment round, the transformation matrix is ​​updated based on the image alignment difference; that is, in the k-th iteration, the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th image alignment difference. This step may include steps 603a-603c (not shown in the figure):

[0117] Step 603a: Perform the k-th round of image transformation on the second contour map based on the k-th transformation matrix to obtain the k-th aligned contour map.

[0118] During the k-th iteration, the computer device performs the k-th image transformation on the second contour map based on the k-th transformation matrix to obtain the transformed k-th aligned contour map. Thus, the k-th increment matrix can be determined based on the difference between the k-th aligned contour map and the first contour map.

[0119] Step 603b: Determine the k-th increment matrix based on the difference between the k-th aligned contour map and the first contour map.

[0120] When measuring image alignment differences, relying solely on the original grayscale values ​​of the image can be inaccurate due to sensitivity to changes in illumination. This embodiment uses the contour values ​​of pixels in the extracted contour maps to measure image alignment differences. This method is relatively insensitive to illumination changes, improving the accuracy of determining the image transformation matrix. Specifically, when two contour maps are perfectly aligned, the image alignment difference is zero. Therefore, minimizing the image difference can be the objective for optimizing the transformation matrix; the optimization objective is:

[0121]

[0122] Optionally, the matrix can be updated using the Gauss-Newton method.

[0123] The image alignment difference can be measured by a second error function. The second error function is shown in the following equation:

[0124]

[0125] Where E0 is the first contour image, and ω(H,E1) represents the transformed image after transforming E1 based on the transformation matrix H.

[0126] In determining the image transformation matrix, an initial transformation matrix H0 is first given. During the k-th iteration, the increment ΔH is searched to find such that... Reaching the minimum value, in When H reaches its minimum value, it can be... k +ΔH is determined as the (k+1)th transformation matrix, and the (k+1)th iteration process begins. Where H... k Let be the k-th transformation matrix. In one possible implementation, the computer device can determine the k-th increment matrix in the k-th iteration process using the difference between the k-th aligned contour map and the first contour map. This process may include the following steps:

[0127] Step 1: Based on the difference between the k-th aligned contour map and the first contour map, determine the value of the second error function for the k-th round. The second error function is used to characterize the image alignment error between the first contour map and the second contour map.

[0128] The computer device calculates based on the k-th aligned contour map and the first contour map. The second error function value for the k-th round is obtained. In one possible implementation, aligned pixels can be determined based on the k-th aligned contour map and the first contour map, and the difference between the corresponding contour values ​​between each aligned pixel can be calculated. Then, the difference can be averaged to obtain the second error function value.

[0129] Step 2: Determine the k-th increment equation based on the second error function value of the k-th round.

[0130] In one possible implementation, the k-th incremental equation can be constructed based on the second error function value of the k-th round. The k-th incremental equation is shown below:

[0131]

[0132]

[0133] Among them, J(H k ) is the Jacobian matrix of the second error function with respect to the parameters. This is the value of the second error function in the kth round.

[0134] Step 3: Determine the k-th increment matrix based on the k-th increment equation.

[0135] The k-th increment matrix can be obtained by solving the k-th increment equation.

[0136] Step 603c: Increment the k-th transformation matrix based on the k-th increment matrix to obtain the (k+1)-th transformation matrix.

[0137] Given the k-th increment matrix, the computer device can perform incremental processing to obtain the (k+1)-th transformation matrix.

[0138] Step 604: If the k-th increment matrix is ​​less than the increment threshold, the (k+1)-th transformation matrix is ​​determined as the image transformation matrix.

[0139] In one possible implementation, when the k-th increment matrix obtained during the iteration process is less than the increment threshold, it is determined that the transformation matrix tends to be stable, that is, it is determined that the iteration termination condition is met. At this time, the (k+1)-th transformation matrix can be determined as the image transformation matrix, and the iteration process is no longer performed.

[0140] Step 605: If the iteration count threshold is reached, the transformation matrix obtained in the last round is determined as the image transformation matrix.

[0141] Alternatively, in another possible implementation, when the number of iterations reaches the iteration threshold, the transformation matrix obtained in the last round can be determined as the image transformation matrix.

[0142] In the above method, the computer device uses the alignment difference of the entire image to measure the accuracy of the transformation matrix. In another possible implementation, instead of using the contrast difference of the entire image pair, only the contrast difference of a local region needs to be determined. The computer device can perform masking on the image and update the transformation matrix based on the contrast difference of the masked contour map.

[0143] Optionally, based on the image alignment result of the kth round after masking, the kth transformation matrix is ​​updated to the (k+1)th transformation matrix, wherein the mask value is determined according to the first position of the pixel in the first contour map and the second position of the pixel mapped to the second contour map.

[0144] The second error function can then be masked, and the masked second error function is as follows:

[0145]

[0146] M is a binary validity mask, where M=0 means that pixel differences are ignored, and M=1 means that the alignment difference corresponding to the pixel needs to be calculated.

[0147] During the masking process, the computer device can determine the mask value corresponding to each pixel based on the local region of interest (ROI). Furthermore, the mask value needs to be determined based on the first position of the pixel in the reference image (first contour map) and the second position of the pixel mapped to the image to be registered (second image).

[0148] That is, M(x,y)=M roi (x,y)·M roi (x′,y′), where (x′,y′)=H -1 ·[x,y,1] T , indicating via H -1 Map (x, y) to the position in the image to be registered, where (x, y) is the pixel position of the pixel in the reference image. If the pixel in the reference image is located within a local region of interest (ROI), and the pixel position mapped to the image to be registered is also located within a local region of interest (ROI), then M = 1. That is, the specific calculation method for the second error function after masking is as follows:

[0149]

[0150] I = E0 - ω(H, E1)

[0151] In each iteration, M is updated based on the latest obtained transformation matrix, and the second error function value is also updated. The transformation matrix is ​​then updated again based on the updated second error function value. That is, in the k-th iteration, the second error function value after mask processing is updated based on the k-th transformation matrix to obtain the k-th increment matrix, and then the (k+1)-th transformation matrix is ​​determined based on the k-th increment matrix.

[0152] Furthermore, in determining the image transformation matrix through image alignment, since directly using the image alignment algorithm requires that the offset between images not be too large, in one possible implementation, the computer device can use a pyramid-style alignment method to alleviate the problem of large image offsets. This method may include the following steps:

[0153] Step 1: Based on the image alignment process of the first contour map and the second contour map at the first resolution, the candidate image transformation matrix is ​​obtained iteratively.

[0154] First, the computer device can align a first contour image and a second contour image at a first resolution, and obtain a candidate image transformation matrix through a multi-round iterative alignment process. Optionally, the first resolution can be half of the original resolution. Furthermore, in the contour image alignment process based on the first resolution, the initial matrix H0 can be an identity matrix.

[0155] Step 2: Image alignment process based on the first contour map and the second contour map at the second resolution. Starting from the candidate image transformation matrix, the image transformation matrix is ​​obtained iteratively. The second resolution is higher than the first resolution.

[0156] After obtaining a preliminary candidate image transformation matrix based on a lower first resolution, the first contour map and the second contour map at a higher second resolution can be aligned in multiple rounds to iterate and obtain the image transformation matrix. In this iteration process, the initial matrix can be the candidate image transformation matrix as the starting point for iteration. Optionally, the second resolution can be the original resolution.

[0157] In the example above, the image transformation matrix is ​​obtained through only two layers of pyramid-style alignment. In practical applications, multi-layer pyramid-style alignment processes can also be used as needed, with each layer's iteration starting from the result of the previous layer. This embodiment does not impose any limitations on this.

[0158] An illustrative example is that the method for obtaining the image transformation matrix based on image alignment can be as follows: Figure 7 As shown. After obtaining the first contour map 701 and the second contour map 702, downsampling can be performed first, that is, adjusted to 1 / 2 resolution. Then, image alignment is performed based on the first contour map and the second contour map at 1 / 2 resolution. The candidate image transformation matrix H1 is obtained according to the Gauss-Newton method. After that, the upsampling process can be performed to restore the original resolution. Then, the image transformation matrix H is obtained iteratively starting from the candidate image transformation matrix H1.

[0159] In this embodiment, the image transformation matrix is ​​obtained iteratively through direct image alignment, and the image alignment difference is measured by the contour value in the contour map during the iteration process, which can improve the accuracy of determining the image alignment difference, thereby improving the accuracy of determining the image transformation matrix.

[0160] Furthermore, in this embodiment, iterative processing based on the image alignment results of local regions can be performed through masking, which helps to improve computational efficiency and thus increase the iteration speed.

[0161] In addition, in this embodiment, the final image transformation matrix is ​​obtained through a pyramid-style iterative alignment process, which avoids the situation where the image transformation matrix calculated due to excessive image offset is inaccurate, thus helping to improve the accuracy of the image transformation matrix.

[0162] The above embodiments describe determining the image transformation matrix using contour point alignment and determining the image transformation matrix using image alignment, respectively. In one possible implementation, the computer device can calculate the image transformation matrix using contour point alignment when the contour is significant. When the contour is relatively blurry, the image transformation matrix is ​​calculated using image alignment. Alternatively, in another possible implementation, after obtaining the first contour image and the second contour image, the computer device can determine two image transformation matrices using the two methods respectively, and then determine the final image transformation matrix based on the mean of the image transformation matrices, thereby improving the accuracy of the image transformation matrix.

[0163] In this embodiment, edge extraction is performed using an edge extraction network. This edge extraction network is trained using a semi-supervised method. The training process of the network will be described below as an example.

[0164] Please refer to Figure 8 This document illustrates a flowchart of a training method for an edge extraction network provided in an exemplary embodiment of this application. This embodiment uses a computer device as an example for illustration, and the method includes the following steps.

[0165] Step 801: Edge detection is performed on the sample image using the Canny algorithm to obtain the label contour map.

[0166] Since the edges extracted by the edge extraction network are relatively coarse, labeled data is required for training. Traditional edge detection algorithms cannot balance the two major requirements of suppressing internal details and maintaining the integrity of the outer contour. Therefore, in this embodiment, the edge detection algorithm and the edge extraction network are combined to improve the accuracy of edge detection.

[0167] In one possible implementation, the computer device can first perform edge detection on the sample image using the Canny algorithm to obtain the contour map of the sample image. Then, the contour map detected by the Canny algorithm can be used as a pseudo-label to train the edge extraction network.

[0168] Furthermore, the contour maps extracted by the Canny algorithm differ depending on the threshold value. Computer devices can use the Canny algorithm with different threshold values ​​to train the edge extraction network at different stages.

[0169] Optionally, edge detection can be performed on the sample image using the Canny algorithm under a first threshold to obtain a first label contour map.

[0170] Optionally, edge detection is performed on the sample image using the Canny algorithm under the second threshold to obtain a second label contour map, where the first threshold is higher than the second threshold.

[0171] That is, the computer equipment uses the Canny algorithm with a high threshold to detect the first label contour map and the Canny algorithm with a low threshold to detect the second label contour map.

[0172] Step 802: Using the label contour map as pseudo-labels, update and train the initial edge extraction network to obtain the candidate edge extraction network.

[0173] In one possible implementation, the process of training the initial edge extraction network using the label contour map detected by the Canny algorithm as pseudo-labels may include steps 802a-802c (not shown in the figure):

[0174] Step 802a: Using the first label contour map as a pseudo label, update and train the initial edge extraction network to obtain the updated edge extraction network.

[0175] In the first stage of training, the initial training is performed using the first labeled contour map as a pseudo-label. That is, edge extraction is performed through the initial edge extraction network to obtain the predicted contour map. The computer device can determine the contrast loss based on the difference between the predicted contour map and the first labeled contour map, and train the initial edge extraction network using the contrast loss. When the loss converges, the first stage of training can be terminated, and the updated edge extraction network can be obtained.

[0176] Step 802b: Perform edge detection on the sample image by updating the edge extraction network to obtain the updated label contour map.

[0177] After the first phase of training, the computer device uses the second label contour map for the second phase of training to improve the completeness of edge extraction network extraction. In this process, the computer device first performs edge detection on the sample images by updating the edge extraction network to obtain an updated label contour map.

[0178] Step 802c: Using the second label contour map and the updated label contour map as pseudo-labels, update and train the updated edge extraction network to obtain the candidate edge extraction network.

[0179] The computer device can then fuse the second label contour map with the updated label contour map to obtain a fused label contour map. This fused label contour map can then be used as a pseudo-label to train the updated edge extraction network, thereby obtaining a candidate edge extraction network and completing the second stage of training.

[0180] The fusion method involves multiplying the contour values ​​of each corresponding pixel in the second label contour map with those in the updated label contour map to obtain the fused label contour map.

[0181] Furthermore, to further enable the edge extraction network to focus on salient edges while ignoring internal details, the sample images can be smoothed to obtain coarse sample images after suppressing high-frequency details. The edge extraction network then performs edge extraction on both the sample image and the coarse sample image, respectively, to obtain a first predicted contour map and a second predicted contour map. The consistency loss is then determined using the first and second predicted contour maps, and the network is trained based on this consistency loss. In other words, during the first and second training stages, the network is trained using the contrast loss and consistency loss determined by the pseudo-labels.

[0182] Step 803: Train the candidate edge extraction network using the teacher edge extraction network to obtain the edge extraction network. The teacher edge extraction network is an edge extraction network trained using real labels.

[0183] In the third stage of training, the edge extraction network trained with real labels can be used as the teacher edge extraction network to train the candidate edge extraction network obtained in the second stage, thus obtaining the final edge extraction network. This can improve the ability to suppress internal details and retain accurate edge localization.

[0184] When training the candidate edge extraction network using the teacher edge extraction network, the computer device extracts edges from the sample images using the teacher edge extraction network to obtain a teacher label contour map, and then extracts edges from the sample images using the candidate edge extraction network to obtain a candidate label contour map. Next, the teacher label contour map and the candidate label contour map are multiplied, and the resulting contour map is used as the pseudo-label in the third stage. This is then used to continue updating and training the candidate edge extraction network, resulting in the final edge extraction network, which is then used in subsequent edge extraction processes.

[0185] In this embodiment, the contour map detected by the edge detection algorithm is used as a pseudo-label to train the edge extraction network. Through a three-stage training process, the edge extraction network can focus on significant edges while ignoring internal details, which can improve the accuracy of edge extraction and thus improve the accuracy of determining the image transformation matrix.

[0186] Figure 9 This is a structural block diagram of an image registration apparatus provided in an exemplary embodiment of this application, such as... Figure 9 As shown, the device includes:

[0187] Image acquisition module 901 is used to acquire a pair of images to be registered, wherein the pair of images includes a first image and a second image;

[0188] The edge extraction module 902 is used to extract edges from the first image and the second image through an edge extraction network to obtain a first contour map and a second contour map.

[0189] The matrix update module 903 is used to perform contour alignment of the first contour map and the second contour map based on the kth transformation matrix during the kth iteration, and to update the kth transformation matrix to the (k+1)th transformation matrix based on the contour alignment result of the kth iteration, wherein the contour alignment difference corresponding to the updated (k+1)th transformation matrix is ​​less than the contour alignment difference corresponding to the kth transformation matrix, and k is a positive integer.

[0190] The matrix determination module 904 is used to determine the transformation matrix obtained in the last round as the image transformation matrix when the iteration termination condition is met. The image transformation matrix is ​​used to perform image registration between the first image and the second image.

[0191] Optionally, the matrix update module 903 is further configured to:

[0192] Based on the k-th transformation matrix, the contour points in the first contour map and the second contour map are aligned in the k-th round, and the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th round point set alignment result;

[0193] And / or,

[0194] Based on the k-th transformation matrix, the first contour map and the second contour map are image aligned in the k-th round, and based on the image alignment result of the k-th round, the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix.

[0195] Optionally, the device further includes:

[0196] The point set determination module is used to filter out a first contour point set from the first contour map and to filter out a second contour point set from the second contour map;

[0197] The matrix update module 903 is also used for:

[0198] Based on the kth transformation matrix, the second contour point set is transformed in the kth round to obtain the kth contour point set;

[0199] Based on the distance between the first set of contour points and each point in the kth set of contour points, determine the kth round of matching contour point pairs;

[0200] Based on the correspondence between the contour point pairs matched in the kth round, the (k+1)th transformation matrix is ​​determined.

[0201] Optionally, the matrix update module 903 is further configured to:

[0202] By using the nearest neighbor search method, the kth contour point that matches the first contour point is obtained from the set of the kth contour points, and the first contour point belongs to the first contour point set.

[0203] If the distance between the first contour point and the kth contour point is less than the distance error threshold, the second contour point corresponding to the first contour point and the kth contour point is determined as a matching contour point pair.

[0204] Optionally, the matrix determination module 904 is further configured to:

[0205] If the first error function converges, the transformation matrix obtained in the last round is determined as the image transformation matrix. The first error function is used to represent the distance error between the first contour point set and the matching contour point pairs in the transformed second contour point set.

[0206] or,

[0207] If the iteration count threshold is reached, the transformation matrix obtained in the last round is determined as the image transformation matrix.

[0208] Optionally, the matrix update module 903 is further configured to:

[0209] Based on the kth transformation matrix, the second contour map is subjected to the kth round of image transformation to obtain the kth aligned contour map;

[0210] Based on the difference between the k-th aligned contour map and the first contour map, the k-th increment matrix is ​​determined;

[0211] Based on the k-th increment matrix, the k-th transformation matrix is ​​incrementally processed to obtain the (k+1)-th transformation matrix.

[0212] Optionally, the matrix update module 903 is further configured to:

[0213] Based on the difference between the kth aligned contour map and the first contour map, the value of the second error function in the kth round is determined. The second error function is used to characterize the image alignment error between the first contour map and the second contour map.

[0214] Based on the second error function value of the kth round, determine the kth increment equation;

[0215] Based on the k-th increment equation, the k-th increment matrix is ​​determined.

[0216] Optionally, the matrix determination module 904 is further configured to:

[0217] If the k-th increment matrix is ​​less than the increment threshold, the (k+1)-th transformation matrix is ​​determined as the image transformation matrix;

[0218] or,

[0219] If the iteration count threshold is reached, the transformation matrix obtained in the last round is determined as the image transformation matrix.

[0220] Optionally, the matrix update module 903 is further configured to:

[0221] Based on the image alignment result of the kth round after mask processing, the kth transformation matrix is ​​updated to the (k+1)th transformation matrix, wherein the mask value is determined according to the first position of the pixel in the first contour map and the second position of the pixel mapped to the second contour map.

[0222] Optionally, the outline alignment method is image alignment;

[0223] The matrix determination module 904 is further configured to:

[0224] Based on the image alignment process of the first contour map and the second contour map at the first resolution, the candidate image transformation matrix is ​​obtained iteratively.

[0225] Based on the image alignment process of the first contour map and the second contour map at the second resolution, the image transformation matrix is ​​obtained iteratively, starting from the candidate image transformation matrix, where the second resolution is higher than the first resolution.

[0226] Optionally, the device further includes:

[0227] The edge detection module is used to detect edges in the sample image using the Canny algorithm to obtain the label contour map;

[0228] The network training module is used to update and train the initial edge extraction network using the label contour map as pseudo-labels to obtain a candidate edge extraction network.

[0229] The network training module is further configured to train the candidate edge extraction network using the teacher edge extraction network to obtain the edge extraction network, wherein the teacher edge extraction network is an edge extraction network trained using real labels.

[0230] Optionally, the edge detection module is further configured to:

[0231] Edge detection is performed on the sample image using the Canny algorithm under the first threshold to obtain the first label contour map;

[0232] Edge detection is performed on the sample image using the Canny algorithm at a second threshold to obtain a second label contour map, wherein the first threshold is higher than the second threshold;

[0233] The network training module is also used for:

[0234] Using the first label contour map as a pseudo label, the initial edge extraction network is updated and trained to obtain the updated edge extraction network.

[0235] The updated edge extraction network is used to perform edge detection on the sample image to obtain an updated label contour map;

[0236] Using the second label contour map and the updated label contour map as pseudo-labels, the updated edge extraction network is updated and trained to obtain the candidate edge extraction network.

[0237] In this embodiment, for the first image and the second image to be registered, the computer device can first perform edge extraction on the images to obtain the first contour map and the second contour map corresponding to the first image, and then perform contour alignment based on the contour maps to obtain the transformation matrix between the first image and the second image. The transformation matrix is ​​determined by contour alignment without relying on the extraction of feature points. Furthermore, by iteratively aligning the contour maps between the first and second contour maps, the image transformation matrix is ​​obtained with the goal of minimizing the contour alignment difference, which can improve the accuracy of estimating the image transformation matrix and help improve the accuracy of image registration.

[0238] It should be noted that the apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their implementation process can be found in the method embodiments, which will not be repeated here.

[0239] Please refer to Figure 10This illustration shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. The computer device can be implemented as a terminal or server in the above embodiments. Specifically, the computer device 1000 includes a Central Processing Unit (CPU) 1001, a system memory 1004 including a random access memory 1002 and a read-only memory 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 that facilitates the transmission of information between various devices within the computer, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0240] In some embodiments, the basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0241] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include computer-readable media (not shown) such as a hard disk or drive.

[0242] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0243] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the methods described above, and the central processing unit 1001 executes the one or more programs to implement the methods provided in the various method embodiments described above.

[0244] According to various embodiments of this application, the computer device 1000 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer systems (not shown).

[0245] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0246] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the image registration method described in any of the above embodiments.

[0247] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image registration method provided above.

[0248] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into the terminal. The computer-readable storage medium stores at least one instruction, at least one program segment, a code set, or an instruction set. The at least one instruction, the at least one program segment, the code set, or the instruction set is loaded and executed by a processor to implement the image registration method described in any of the above method embodiments.

[0249] Optionally, the computer-readable storage medium may include ROM, RAM, solid-state drives (SSDs), or optical discs, etc. The RAM may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments described above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0250] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0251] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image registration method characterized by, The method includes: Obtain the registration image pair to be registered, wherein the registration image pair includes a first image and a second image; Edge extraction is performed on the first image and the second image using an edge extraction network to obtain a first contour map and a second contour map; In the k-th iteration, when the contours are significant, the contour points in the first and second contour maps are aligned using the k-th transformation matrix, and the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th point set alignment result. The (k+1)-th transformation matrix is ​​related to the distance between each point in the first and k-th contour point sets. When the contours are blurred, the first and second contour maps are aligned using the k-th transformation matrix, and the k-th transformation matrix is ​​updated to the (k+1)-th transformation matrix based on the k-th image alignment result. The (k+1)-th transformation matrix is ​​related to the difference between the k-th aligned contour map and the first contour map. The contour alignment difference corresponding to the updated (k+1)-th transformation matrix is ​​less than the contour alignment difference corresponding to the k-th transformation matrix, where k is a positive integer. If the iteration termination condition is met, the transformation matrix obtained in the last round is determined as the image transformation matrix, which is used to perform image registration between the first image and the second image.

2. The method of claim 1, wherein, The method further includes: A first set of contour points is obtained by filtering from the first contour map, and a second set of contour points is obtained by filtering from the second contour map; The step of performing k-th round point set alignment on the contour points in the first contour map and the second contour map based on the k-th transformation matrix, and updating the k-th transformation matrix to the (k+1)-th transformation matrix based on the k-th round point set alignment result, includes: Based on the kth transformation matrix, the second contour point set is transformed in the kth round to obtain the kth contour point set; Based on the distance between the first set of contour points and each point in the kth set of contour points, determine the kth round of matching contour point pairs; Based on the correspondence between the contour point pairs matched in the kth round, the (k+1)th transformation matrix is ​​determined.

3. The method of claim 2, wherein, The step of determining the k-th round matching contour point pair based on the distance between the first contour point set and each point in the k-th contour point set includes: By using the nearest neighbor search method, the kth contour point that matches the first contour point is obtained from the set of the kth contour points, and the first contour point belongs to the first contour point set. If the distance between the first contour point and the kth contour point is less than the distance error threshold, the second contour point corresponding to the first contour point and the kth contour point is determined as a matching contour point pair.

4. The method of claim 2, wherein, The step of determining the transformation matrix obtained in the last round as the image transformation matrix when the iteration termination condition is met includes: If the first error function converges, the transformation matrix obtained in the last round is determined as the image transformation matrix. The first error function is used to represent the distance error between the first contour point set and the matching contour point pairs in the transformed second contour point set. or, If the iteration count threshold is reached, the transformation matrix obtained in the last round is determined as the image transformation matrix.

5. The method of claim 1, wherein, The step of performing k-th round image alignment on the first contour map and the second contour map based on the k-th transformation matrix, and updating the k-th transformation matrix to the (k+1)-th transformation matrix based on the k-th round image alignment result, includes: Based on the kth transformation matrix, the second contour map is subjected to the kth round of image transformation to obtain the kth aligned contour map; Based on the difference between the k-th aligned contour map and the first contour map, the k-th increment matrix is ​​determined; Based on the k-th increment matrix, the k-th transformation matrix is ​​incrementally processed to obtain the (k+1)-th transformation matrix.

6. The method of claim 5, wherein, The step of determining the k-th increment matrix based on the difference between the k-th aligned contour map and the first contour map includes: Based on the difference between the kth aligned contour map and the first contour map, the value of the second error function in the kth round is determined. The second error function is used to characterize the image alignment error between the first contour map and the second contour map. Based on the second error function value of the kth round, determine the kth increment equation; Based on the k-th increment equation, the k-th increment matrix is ​​determined.

7. The method of claim 5, wherein, The step of determining the transformation matrix obtained in the last round as the image transformation matrix when the iteration termination condition is met includes: If the k-th increment matrix is ​​less than the increment threshold, the (k+1)-th transformation matrix is ​​determined as the image transformation matrix; or, If the iteration count threshold is reached, the transformation matrix obtained in the last round is determined as the image transformation matrix.

8. The method of claim 1, wherein, The step of updating the k-th transformation matrix to the (k+1)-th transformation matrix based on the k-th round image alignment result includes: Based on the image alignment result of the kth round after mask processing, the kth transformation matrix is ​​updated to the (k+1)th transformation matrix, wherein the mask value is determined according to the first position of the pixel in the first contour map and the second position of the pixel mapped to the second contour map.

9. The method of claim 1, wherein, The contour alignment method is image alignment, and the method further includes: Based on the image alignment process of the first contour map and the second contour map at the first resolution, the candidate image transformation matrix is ​​obtained iteratively. Based on the image alignment process of the first contour map and the second contour map at the second resolution, the image transformation matrix is ​​obtained iteratively, starting from the candidate image transformation matrix, where the second resolution is higher than the first resolution.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Edge detection is performed on the sample images using the Canny algorithm to obtain the label contour map; Using the label contour map as pseudo-labels, the initial edge extraction network is updated and trained to obtain a candidate edge extraction network; The candidate edge extraction network is trained by the teacher edge extraction network to obtain the edge extraction network, which is an edge extraction network trained using real labels.

11. The method according to claim 10, characterized in that, The step of performing edge detection on the sample image using the Canny algorithm to obtain the label contour map includes: Edge detection is performed on the sample image using the Canny algorithm under the first threshold to obtain the first label contour map; Edge detection is performed on the sample image using the Canny algorithm at a second threshold to obtain a second label contour map, wherein the first threshold is higher than the second threshold; The step of updating and training the initial edge extraction network using the label contour map as pseudo-labels to obtain a candidate edge extraction network includes: Using the first label contour map as a pseudo label, the initial edge extraction network is updated and trained to obtain the updated edge extraction network. The updated edge extraction network is used to perform edge detection on the sample image to obtain an updated label contour map; Using the second label contour map and the updated label contour map as pseudo-labels, the updated edge extraction network is updated and trained to obtain the candidate edge extraction network.

12. An image registration device, characterized in that, The device includes: The image acquisition module is used to acquire a pair of images to be registered, wherein the pair of images includes a first image and a second image; An edge extraction module is used to extract edges from the first image and the second image through an edge extraction network to obtain a first contour map and a second contour map. The matrix update module is used to perform point set alignment of the contour points in the first contour map and the second contour map based on the k-th transformation matrix during the k-th iteration. If the contours are significant, the module updates the k-th transformation matrix to the (k+1)-th transformation matrix based on the k-th point set alignment result. The (k+1)-th transformation matrix is ​​related to the distance between the points in the first contour point set and the k-th contour point set. If the contours are blurred, the module performs image alignment of the first contour map and the second contour map based on the k-th transformation matrix. The module updates the k-th transformation matrix to the (k+1)-th transformation matrix based on the k-th image alignment result. The (k+1)-th transformation matrix is ​​related to the difference between the k-th aligned contour map and the first contour map. The updated contour alignment difference corresponding to the (k+1)-th transformation matrix is ​​less than the contour alignment difference corresponding to the k-th transformation matrix, where k is a positive integer. The matrix determination module is used to determine the transformation matrix obtained in the last round as the image transformation matrix when the iteration termination condition is met. The image transformation matrix is ​​used to perform image registration between the first image and the second image.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the image registration method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one program, which is loaded and executed by a processor to implement the image registration method as described in any one of claims 1 to 11.

Images