Large-scale deformation compensation method, device, equipment and medium

Abstract

The invention discloses a large-scale deformation compensation method, device, equipment and medium, and relates to the field of image processing, and the method regards a salient point in an image as a leader in a group, is similar to a leading animal in a migration sheep flock, and plays a key role in guiding the whole registration process. In order to estimate the aggregation manifold, large-scale deformation registration is divided into three parts according to an animal migration mode: route decision based on aggregation quantization, route execution with aggregation maintenance and cascade migration with aggregation inheritance. The three parts are integrated into an aggregation cascade migration (CCM) framework, and large-scale deformation of the image is effectively compensated. According to the method, the registration precision and robustness during large-scale deformation compensation can be greatly improved. The method is a universal method, is not limited to specific feature descriptors or deformation models, and can be easily expanded to other image registration tasks.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a large-scale deformation compensation method, device, equipment and medium based on focal registration. Background Art

[0002] Image registration is the process of matching and superimposing two or more images acquired at different times, with different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.). It has been widely used in remote sensing data analysis, computer vision, image processing and other fields.

[0003] Image registration and correlation is a typical problem and technical difficulty in the field of image processing research. Its purpose is to compare or fuse images of the same object acquired under different conditions. For example, the images may come from different acquisition devices, taken at different times, different shooting angles, etc. Sometimes, image registration problems for different objects are also needed.

[0004] Specifically, for two images in a set of image data, a spatial transformation is found to map one image (moving image) to another image (reference image, fixed image), so that the points corresponding to the same position in the two images are matched one by one, thereby achieving the purpose of information fusion. The most widely used registration methods include image registration methods based on grayscale information, etc. and registration methods based on feature point matching.

[0005] However, the traditional registration method based on image grayscale similarity statistics has many disadvantages. For example, when faced with large-scale deformation, the objective function often becomes non-convex because the role of salient points is not considered. The use of traditional small update steps (usually used to optimize the objective function to ensure convergence stability) may cause the algorithm to fall into a local minimum. The traditional registration method based on feature point matching also has many disadvantages. For example, it is difficult to locate reliable feature points, and it is a challenge to balance the number and quality of feature points.

[0006] It can be seen that when facing large-scale deformation, due to the lack of information on the relative displacement between pixels, the existing technology may not be able to accurately compensate for the deformation. Summary of the invention

[0007] In view of the above problems, the present invention provides a large-scale deformation compensation method, device, equipment and medium for overcoming the above problems or at least partially solving the above problems.

[0008] The present invention provides the following scheme:

[0009] A large-scale deformation compensation method, comprising:

[0010] Extracting features from the fixed image and the floating image to obtain feature extraction results, and using a feature matching method and a parameter ratio test threshold in combination with the feature extraction results to identify and obtain a number of candidate routes, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image;

[0011] A route consistency quantification module is used to quantify the consistency of the candidate routes to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes;

[0012] Using an agglomerative clustering algorithm to determine a number of agglomerative routes from the consistency matrix;

[0013] A route checking strategy is used to determine a final collective route from a plurality of the clustered routes, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes;

[0014] Using thin plate spline interpolation of the final collective route to obtain a flight field for guiding the group, the flight field is a large-scale dense deformation field;

[0015] A diffusion field is obtained by using a route stop strategy in combination with the flight field estimation, wherein the diffusion field is a small-scale dense deformation field;

[0016] The flight field and the diffusion field are deformed to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to compensate for large-scale deformation and small-scale deformation of the floating image during the deformation process;

[0017] The floating image is deformed continuously through multiple migrations using a cascade migration framework and the migration field until a target deformed image is obtained after the migration converges.

[0018] Preferably: the loosest ratio test is used to determine all feature points with bijective matching as the candidate routes.

[0019] Preferably: the distance consistency is represented by the following formula:

[0020]

[0021] Where: k dis ∈(0,+∞] is used to control the sensitivity of distance consistency d(i,j), d m Represents the relative distance in the floating image, d fRepresents the relative distance in a fixed image;

[0022] The Avalon-based voting method is used to quantify the topological consistency, which is expressed by the following formula:

[0023] t(i,j)=t(i)·t(j)

[0024] Where: t(i) represents route v i The route-level topology of , t(j) represents the route v j The route level topology of route v j Indicates route v i Neighborhood routes.

[0025] Preferably: t(i) is represented by the following formula:

[0026]

[0027] Where: k top ∈(0,+∞] is used to control the sensitivity of topology t(i), J rr (v i ) indicates v i Relative to its neighbors Topological changes;

[0028] The J rr (v i ) is represented by the following formula:

[0029]

[0030] Where: J without (v i ) and J with (v i ) respectively represent the introduction of v i Before and after topology.

[0031] Preferably: the final collective route is represented by the following formula:

[0032]

[0033] Where: c i Represents Ω i The contribution of the i-th collective route in Ω i represents the i-th region of interest.

[0034] Preferably: the route stop strategy collective route The drift distance is used as the penalty term in the objective function and is expressed as follows:

[0035]

[0036] Where: represents the normalized cross-correlation similarity, Represents a route stop item, defined as k rs represents the parameter that weighs similarity and penalty, Indicates the flight field, represents the diffusion field.

[0037] Preferably: before the cascade migration converges, the current collective route is merged into the previous collective route In the formula, it is expressed as follows:

[0038]

[0039] Where: Indicates the current collective route, Indicates the previous collective route.

[0040] A large-scale deformation compensation device, characterized in that it is used to perform the above-mentioned large-scale deformation compensation method, and the device comprises:

[0041] A candidate route identification unit is used to extract features from a fixed image and a floating image to obtain feature extraction results, and to identify and obtain a number of candidate routes by combining the feature extraction results with a feature matching method and a parameter ratio test threshold, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image;

[0042] A consistency matrix acquisition unit, used to quantify the consistency of the candidate routes using a route consistency quantification module to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes;

[0043] A clustering route determination unit, used to determine a number of clustering routes from the consistency matrix using a clustering algorithm;

[0044] A final collective route determination unit is used to determine a final collective route from the plurality of clustered routes using a route checking strategy, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes;

[0045] A flight field acquisition unit, used for obtaining a flight field for guiding the group by using thin plate spline interpolation of the final collective route, wherein the flight field is a large-scale dense deformation field;

[0046] A diffusion field acquisition unit, used to obtain a diffusion field by using a route stop strategy in combination with the flight field estimation, wherein the diffusion field is a small-scale dense deformation field;

[0047] A migration field acquisition unit is used to perform deformation field compounding of the flight field and the diffusion field to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to perform large-scale deformation compensation and small-scale deformation compensation on the floating image during the deformation process;

[0048] The cascade migration unit is used to continuously deform the floating image through multiple migrations using the cascade migration framework and the migration field until a target deformed image is obtained after the migration converges.

[0049] A large-scale deformation compensation device, the device comprising a processor and a memory:

[0050] The memory is used to store program code and transmit the program code to the processor;

[0051] The processor is used to execute the above-mentioned large-scale deformation compensation method according to the instructions in the program code.

[0052] A computer-readable storage medium is used to store program codes, and the program codes are used to execute the above-mentioned large-scale deformation compensation method.

[0053] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0054] The embodiments of the present application provide a large-scale deformation compensation method, device, equipment and medium. The method regards the salient points in the image as leaders in the group, similar to the leading animals in the migrating flock of sheep, and plays a key role in guiding the entire registration process. In order to estimate the aggregation manifold, the large-scale deformation registration is divided into three parts according to the migration pattern of the animal: route decision based on aggregation quantification, route execution with aggregation preservation, and cascade migration with aggregation inheritance. These three parts are integrated into the aggregation cascade migration (CCM) framework, which effectively compensates for the large-scale deformation of the image. This method can greatly improve the registration accuracy and robustness when compensating for large-scale deformations. This method is a general method that is not limited to specific feature descriptors or deformation models and can be easily extended to other image registration tasks. Because this method uses the concept of aggregation motion to provide new ideas for large-scale deformation problems, it will bring more scalable space to the field of image registration.

[0055] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 is a flow chart of a large-scale deformation compensation method provided by an embodiment of the present invention;

[0058] Figure 2 is a framework diagram of a large-scale deformation compensation method provided by an embodiment of the present invention;

[0059] Figure 3 is a schematic diagram of distance consistency provided by an embodiment of the present invention;

[0060] Figure 4 It is a schematic diagram of high directional consistency and high distance consistency provided by an embodiment of the present invention;

[0061] Figure 5 is a voting flow chart based on Avalon provided by an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of a large-scale deformation compensation device provided by an embodiment of the present invention;

[0063] Figure 7 Schematic diagram of a large-scale deformation compensation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0065] See also Figure 1 , is a large-scale deformation compensation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:

[0066] S101: Perform feature extraction on a fixed image and a floating image to obtain feature extraction results, and use a feature matching method and a parameter ratio test threshold in combination with the feature extraction results to identify a number of candidate routes, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image. In specific implementation, the embodiment of the present application can provide the use of the loosest ratio test to determine all feature points with bijective matching as the candidate routes.

[0067] S102: Using a route consistency quantification module to quantify the consistency of the candidate routes to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes; in specific implementation, the embodiment of the present application can provide that the distance consistency is expressed by the following formula:

[0068]

[0069] Where: k dis ∈(0,+∞] is used to control the sensitivity of distance consistency d(i,j), d m Represents the relative distance in the floating image, d f Represents the relative distance in a fixed image;

[0070] The Avalon-based voting method is used to quantify the topological consistency, which is expressed by the following formula:

[0071] t(i,j)=t(i)·t(j)

[0072] Where: t(i) represents route v i The route-level topology of , t(j) represents the route v j The route level topology of route v j Indicates route v i Neighborhood routes.

[0073] t(i) is expressed by the following formula:

[0074]

[0075] Where: k top ∈(0, +∞] is used to control the sensitivity of topology t(i), J rr (vi) means v i Relative to its neighbors Topological changes;

[0076] The J rr (v i ) is represented by the following formula:

[0077]

[0078] Where: J without (v i ) and J with (v i ) respectively represent the introduction of v i Before and after topology.

[0079] S103: using an agglomerative clustering algorithm to determine a number of agglomerative routes from the consistency matrix;

[0080] S104: A route checking strategy is used to determine a final collective route from the plurality of clustered routes, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes; in specific implementation, the embodiment of the present application may provide that the final collective route is represented by the following formula:

[0081]

[0082] Where: c i Represents Ω i The contribution of the i-th collective route in Ω i represents the i-th region of interest.

[0083] S105: using thin plate spline interpolation of the final collective route to obtain a flight field for guiding the group, wherein the flight field is a large-scale dense deformation field;

[0084] S106: Using the route stop strategy in combination with the flight field estimation to obtain a diffusion field, the diffusion field is a small-scale dense deformation field; in specific implementation, the embodiment of the present application can provide the route stop strategy collective route The drift distance is used as the penalty term in the objective function and is expressed as follows:

[0085]

[0086] Where: represents the normalized cross-correlation similarity, Represents a route stop item, defined as k rs represents the parameter that weighs similarity and penalty, Indicates the flight field. represents the diffusion field.

[0087] S107: Compounding the flight field and the diffusion field into a deformation field to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to perform large-scale deformation compensation and small-scale deformation compensation on the floating image during the deformation process;

[0088] S108: Using a cascade migration framework and the migration field to continuously deform the floating image through multiple migrations until a target deformed image is obtained after the migration converges.

[0089] Furthermore, before the cascade migration converges, the current collective route is merged into the previous collective route. In the formula, it is expressed as follows:

[0090]

[0091] Where: Indicates the current collective route, Indicates the previous collective route.

[0092] The large-scale deformation compensation method provided by the embodiment of the present application models large deformation compensation as a clustering movement, which includes three main components: route decision based on clustering quantification, route execution with clustering preservation, and cascade migration with clustering inheritance. In the first component, given fixed and floating images, a traditional feature matching method is used to identify all correspondences as candidate routes with a loose ratio test threshold.

[0093] Subsequently, distance consistency and topological consistency are used to quantify the consistency of these routes in combination with directional consistency. An efficient Avalon-based voting method is designed to calculate topological consistency. Based on the consistency matrix, route clustering is applied to derive collective routes. Finally, a route checking strategy is proposed to decide the final route by excluding negative contribution routes, demonstrating the ability of driving images for large deformation compensation.

[0094] In the second component, the TPS interpolation of the final route is used to obtain the large-scale dense deformation field of the leading group, called flight. Then, under the initialization of the flight, the small-scale dense deformation field of the image grayscale is estimated, called diffusion. In order to maintain the clustering of the route, a route stay strategy is proposed to keep the leader point on the original route while making the group diffuse. Flight and diffusion constitute a migration that promotes the transformation of the floating image.

[0095] In the third component, a cascade migration framework with a route merging strategy is introduced to transfer the current route to the next migration, ensuring the consistency between them. When a new route cannot be identified in the current iteration, the framework converges, resulting in the final deformed image.

[0096] The large-scale deformation compensation method provided by this application is introduced in detail below.

[0097] like Figure 2 As shown, after quantifying and maintaining the clustering of routes, collective migration promotes smooth transformations and compensates for large-scale and small-scale deformations. Multiple migrations are integrated into a cascade migration framework until convergence, i.e., no new routes can be identified in the current iteration.

[0098] Route decision making based on clustering quantification.

[0099] Candidate routes identified.

[0100] Define candidate routes as feature points p in fixed image f and floating image mf and p m The potential correspondence between them can be identified by feature matching methods. In this application, FAST+BRISK+mutual nearest neighbor search is used to identify candidate routes. This method can quickly and effectively extract and describe features on various data sets without the need for a large number of annotated data sets for training. For 3D data sets, extending FAST and BRISK to 3D scenes is a trivial extension. This extension can be efficiently implemented through parallel computing technology on GPUs.

[0101] The parameter ratio test (r) controls the number of candidate routes. In this application, the most relaxed ratio test (r=1) is used to include all feature points with bijective matches as candidate routes. This approach is in contrast to many landmark-based methods, as they require a guaranteed match accuracy, typically using only the top 1% of matches and random sampling consensus (RANSAC) techniques, otherwise the accuracy of the estimated deformation field drops dramatically. In this application, correct matches are extracted based on the clustering of routes, which allows all matches to be used as input without reducing precision, on the contrary, accuracy will be improved due to the increased recall of the matches.

[0102] Route consistency quantification.

[0103] After determining the candidate routes, we can get the leader point set P f and its destination set P m , and the set of routes between them V = {p m -p f |p f ∈P f ,p m ∈P m}. The number of candidate routes is recorded as N R =|P f |=|P m |=|V|. In this application, the i-th leader point is represented as According to its route v i , it can reach its destination The neighbor leader point of in is a fixed image domain In this application, K=50 is set for all datasets. i The neighbor route of in Depend on definition.

[0104] To quantify the clustering of V, we start with the route consistency in the neighborhood, where route consistency is defined as v iand v j The directional correlation between. This metric only considers the direction of the route and is applicable in cases where the scale of motion is relatively small, such as the optical flow between video frames. However, in large deformations in image registration, although there is a high directional consistency in the large-scale motion of the route, there may be a large relative distance change before and after the motion, resulting in a low distance consistency, as shown in Figure 3 . A low distance consistency indicates severe elastic deformation, which may lead to image distortion. To quantify the elasticity of the deformation, the distance consistency is proposed as follows.

[0105] Having only high directional consistency does not necessarily guarantee high distance consistency. The arrows indicate the direction of the route, and the dashed lines represent the relative distances between the reference leading points and their neighbors. Although high directional consistency is achieved, indicating that the route directions are consistent, the relative distances between the reference leading points and their neighbors have changed significantly, resulting in a low distance consistency.

[0106] Given v i and v j , the relative distance in the fixed image is defined as Similarly, based on the destination and , the relative distance in the floating image is defined as Define v i and v j The distance consistency is as follows.

[0107]

[0108] where k dis ∈(0,+∞] controls the sensitivity of the distance consistency d(i,j). The closer k dis is to 0, the higher the penalty for d(i,j). The overall elasticity can be controlled by k dis . A larger k dis allows for a larger non-rigid deformation, while a smaller k dis will limit it to a rigid deformation.

[0109] The distance consistency limits the amount of relative distance change before and after the motion, but this metric does not strictly guarantee topology. As shown in Figure 4 , although the minimum relative distance change is achieved, the reference leading point goes beyond the solid line range after the motion, resulting in a low topological consistency. A makeshift solution is to use a smaller k dis to make the entire deformation field rigid, but this cannot compensate for non-rigid deformations. The Jacobian determinant (J) can represent the local topological preservation of the field, including dilation (J>1), incompressibility (J = 1), contraction (0<J<1), and folding (J≤0). In the field of finite element analysis, the Jacobian ratio (Jr ) is usually used to calculate the maximum value J of all nodes max With the minimum value J min The ratio of the values ​​of max and J min Close, then J r Converges to 1, indicating that the element is similar to the initial unit element; if there is a negative J r , it means that the element is collapsed and the element fails the shape test. r It has been shown to be effective in finite element analysis, but it cannot be directly applied to route consistency analysis because it measures the topology of the entire element rather than the topology at the route level.

[0110] Inspired by the Jacobian ratio, the route Jacobian ratio (J rr ) to quantify each route v i Its neighbor set To achieve this, we first define J without (v i ) and J with (v i )as follows.

[0111]

[0112] where J(p;TPS(V)) is the Jacobian of the transformation of the TPS technique interpolating at point p using a given set of routes V. without (v i ) and J with (v i ) respectively represent the introduction of v i Before and after topology.

[0113] like Figure 4 As shown in Figure 1, high directional consistency and high distance consistency do not necessarily guarantee high topological consistency. The arrow indicates the direction of the route, and the dashed and solid lines indicate the relative distance and topological structure between the reference leader point and its neighbors, respectively. Despite achieving high directional consistency and distance consistency, the reference leader point is beyond the solid line range after the movement, resulting in low topological consistency.

[0114] Given J without (v i ) and J with (v i ), the route Jacobian ratio is defined as follows.

[0115]

[0116] Among them, J rr (vi ) indicates v i Relative to its neighbors A voting method based on Avalon is proposed to efficiently calculate J rr (v i ), which will be introduced in the next subsection.

[0117] Based on J rr (v i ), a route-level topology is proposed as shown below.

[0118]

[0119] Among them, k top ∈(0,+∞] controls the sensitivity of topology t(i). k top The closer to 0, the higher the rr The further away from 1, the lower t(i) will be.

[0120] After computing the topology of all routes in V, define v i and v j The topological consistency is as follows:

[0121] t(i,j)=t(i)·t(j)

[0122] Based on Avalon's vote:

[0123] J rr (v i ) depends on J without (v i ) and J with (v i ). If J without (v i ) is positive, indicating the introduction of v i The previous topology is good and can be calculated by J with (v i ) to easily isolate v i Contribution. with (v i ) indicates the introduction of v i The topology still remains, indicating that v i is a good topological route, leading to J rr (v i ) is positive. On the contrary, the negative J with (v i ) indicates that the topology is destroyed, and v i Marked as a bad topological route, resulting in J rr (v i ) is negative. However, when J without (v i) is negative (indicates the introduction of v i The previous topology was poor), determine v i The impact on topology becomes challenging. Therefore, determining the route v i The key to the topological quality of is to isolate its contribution, which involves choosing appropriate neighbor routes and calculating the introduction of v i Before and after topology changes.

[0124] This process is very similar to the board game Resistance: Avalon, a popular hidden identity game with two teams: good guys and bad guys. At the start of the game, players are randomly assigned to the good or bad team. In each round, a subset of players is selected to go on a mission. These players privately vote on whether the mission succeeds or fails. If all players vote to succeed, the mission succeeds; otherwise, the mission fails. In a candid game of Avalon, the good guys always vote to succeed, and the bad guys always vote to fail. In order to isolate the contribution of the To Be Confirmed (TBC) players in the subset, the most effective strategy is to introduce only one TBC player per vote, while the other players have already been confirmed as good guys. If the mission fails, the TBC player is identified as the bad guy; otherwise, the TBC player is identified as the good guy.

[0125] Inspired by the Avalon game, we propose Avalon-based voting to identify good routes in an iterative manner. Let V Good 、V Evil and V TBC are the sets of good, bad and unconfirmed routes respectively. These sets satisfy V Good ∪V Evil ∪V TBC =V and In the first iteration, from At the beginning, all routes are unverified. For the xth iteration, traverse To identify each v i This includes three steps: selecting neighbor routes, calculating consistency changes, and determining route identity. The flowchart of this process is shown in Figure 5 shown.

[0126] Based on the voting process of Avalon, for the xth iteration, traverse Each route in is identified by the contribution of isolated routes, including selecting neighbor routes, calculating consistency changes, and determining route identities. The identity of each route is given in Update in, and the iteration continues until and There is no change in between.

[0127] Select Neighborhood Route:

[0128] According to Avalon Gaming Strategy, by excluding All the bad guys in the game and keep the good guys to isolate v i First, choose a neighbor route that maintains the neighbors The leader point remains the K nearest neighbors before and after the movement. Neighbor route v in j Meet the following two conditions: 1. v j The starting point is a fixed image domain Neighbors, 2, v j The destination is the floating image domain Neighbors. As v i The set of K nearest neighbor routes, where the destination of the route is in the floating image domain Neighbors of . Maintain a subset of neighbors The set of K nearest neighbor routes and The intersection definition is:

[0129]

[0130] in, yes and The bijection between . and Determined according to each image domain f and m respectively.

[0131] Then, from Select a subset from as the anchor neighbor route, denoted as Serves as an anchor reference to quantify consistency changes. A good route in, if possible, form Specifically, if The number of good routes in meets the minimum requirement of TPS interpolation, that is, n+1. A good route to build Otherwise, merge the TBC set into the Good set to build

[0132] Calculate consistency change: Given Neighbor routes in J without (v i ) and J with (v i ) to quantify v i The consistency of the route relative to its neighbors. Equation J without (v i ) and J with (v i ) is updated as follows:

[0133]

[0134] Determine route identity: After completing the selection of neighbor routes and calculating the consistency change, determine v based on the results of these two steps. i The criteria for determining identity include the following three situations:

[0135] Good identity: The number of routes in is at least n+1, which indicates the minimum number of routes required to satisfy TPS interpolation. without (v i ) and J with (v i ) are all positive, indicating that the introduction of v i After that, the consistency is well maintained. In this case, v i Mark as good by updating and Then quantify J rr (v i ).

[0136] Bad identity: The number of routes in is less than n+1, indicating that there are not enough routes for TPS interpolation. Or, J without (v i ) is positive, but J with (v i ) is negative, indicating that the introduction of v i If any of the above conditions are met, v i Mark as bad, update by and Then punish J rr (v i )=0.

[0137] To be determined: The number of routes in is at least n+1. However, J without (v i ) is negative, indicating the introduction of v i The consistency before was low. In this case, v i The identity of remains pending until the next iteration.

[0138] After traversing all v i Afterwards, through and Update the set of good, bad, and pending routes. When V TBC When there is no change, All routes in are marked as bad, and the corresponding All are penalized to 0, and the iteration ends.

[0139] Route decision and route checking strategy

[0140] After completing the integration of distance consistency, topological consistency and direction consistency, the specific route γ L The consistency on the route is updated as follows:

[0141]

[0142] Through the clustering of individuals, the consistency matrix Z can be calculated, which represents the route consistency between any two routes. Given a threshold k thr , connections with low route consistency, i.e. Z(i,j) <k thr , are removed by setting Z(i,j) = 0. Then, collective clustering is applied by searching for connected components exceeding a threshold Z as collective routes N C The number of clusters representing the collective routes.

[0143] In this application, all datasets are set in

[0144] Inspired by the fact that appropriate migration routes guide animals to more suitable environments, a route checking strategy is proposed to determine the final route. This strategy measures the enhancement of local image similarity, enabling the identification and elimination of routes with negative contributions. To achieve this, the convex region of each collective route is calculated to obtain the region of interest (ROI) By solving the TPS matrix equation in parallel on the GPU, the GPU-based TPS technology is implemented, so that the displacement field of each ROI can be estimated efficiently. The contribution of each collective route is defined as follows.

[0145]

[0146] Among them, c i is Ω i The contribution of the ith collective route in . Sim(;) represents the similarity measure of two ROIs, which is defined as neighborhood correlation in this application.

[0147] By merging c i The clustering of positive collective routes gives the final collective route, which can drive the entire image to perform large-scale deformation compensation:

[0148]

[0149] Maintaining collective route execution:

[0150] Given the final collective route Route execution begins with interpolating a dense deformation field using TPS techniques, called flight flight Driven by collective routes and have the ability to effectively compensate for large-scale deformations.

[0151] After completing the large-scale compensation, there are still local small-scale deformations present in the image, which can be easily estimated by traditional deformation models based on grayscale information. However, the grayscale of the image may change significantly due to factors such as contrast agents and illumination changes. This may lead to image distortion when optimizing a similarity measure that purely relies on the grayscale information of the image, thus compromising the collectivity of the routes.

[0152] To address this problem, a route-staying strategy is proposed that aims to estimate a deformation field that preserves collectivity, called diffusion Diffusion helps keep the leader point on its original route while allowing the group to spread. The drift distance of is used as the penalty term in the objective function as shown below.

[0153]

[0154] in, It is the normalized cross-correlation similarity, which is robust to overall grayscale changes. is the proposed route stop term, defined as To punish The drift distance of the image can further enhance the robustness to local grayscale changes. rs The similarity and penalty are weighed and set to 0.01 for all datasets. In this application, B-Spline transformation is used as The deformation model is used because of its efficiency.

[0155] By optimizing the objective function Able to use gradient descent techniques to estimate to compensate for local small-scale deformation. Then, and The combination gets migrated Large-scale and small-scale deformations are compensated in turn.

[0156] Cascading migration with collective inheritance

[0157] Unlike traditional methods, which usually converge to a local optimal solution after an initial transformation and non-rigid deformation, our method has the ability to further optimize through cascaded migrations. It is worth noting that the completion of each migration enhances image similarity and helps identify candidate routes. This in turn makes it possible to generate the next migration to drive the next collective motion. To facilitate multiple migrations, a cascaded migration framework with a route merging strategy is introduced.

[0158] Similar to the multiple migrations that animals make to reach their destination, the cascade migration framework involves continuously deforming the floating image m through multiple migrations. Let k cas is the maximum number of migrations, and the yth migration is expressed as The y-th m is defined as Cascade migration from m 0 = m, when the collective route cannot be determined The convergence condition of migration ensures that increasing k cas The registration accuracy can be improved without significantly increasing the computation time. In this application, k is set for all datasets. cas =3.

[0159] In cascading migrations, ensuring the collective nature of the route is crucial. After that, most of the large-scale deformations are effectively compensated, leaving only a small amount of scattered deformations distributed in small areas of the image. The flight generated from these small areas This may cause image drift because the previous migration The route in Considered.

[0160] In order to solve this problem, a route merging strategy is proposed by combining the current collective route Merge into the previous group route Update from Zhonglai:

[0161]

[0162] Through the route merging strategy, the collective nature of the routes in the cascade migration is ensured, which improves the robustness and accuracy of the registration. Once the cascade migration converges, the floating image m is deformed Aligned with the fixed image f in height.

[0163] In summary, the large-scale deformation compensation method provided by the present application regards the salient points in the image as leaders in the group, similar to the leading animal in a migrating flock of sheep, and plays a key role in guiding the entire registration process. In order to estimate the clustering manifold, the large-scale deformation registration is divided into three parts according to the migration pattern of the animal: route decision based on clustering quantification, route execution with clustering preservation, and cascade migration with clustering inheritance. These three parts are integrated into the clustering cascade migration (CCM) framework to effectively compensate for the large-scale deformation of the image. This method can greatly improve the registration accuracy and robustness when compensating for large-scale deformations. This method is a general method that is not limited to specific feature descriptors or deformation models and can be easily extended to other image registration tasks. Because this method uses the concept of clustering motion to provide new ideas for large-scale deformation problems, it will bring more scalable space to the field of image registration.

[0164] See also Figure 6 , the embodiment of the present application can also provide a large-scale deformation compensation device, such as Figure 6 As shown, for performing the above-mentioned large-scale deformation compensation method, the device may include:

[0165] The candidate route identification unit 601 is used to extract features from the fixed image and the floating image to obtain feature extraction results, and use a feature matching method and a parameter ratio test threshold to identify and obtain a number of candidate routes in combination with the feature extraction results, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image;

[0166] A consistency matrix acquisition unit 602 is used to quantify the consistency of the candidate routes using a route consistency quantification module to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes;

[0167] A clustering route determining unit 603, configured to determine a number of clustering routes from the consistency matrix using a clustering algorithm;

[0168] A final collective route determination unit 604 is used to determine a final collective route from the plurality of clustered routes using a route checking strategy, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes;

[0169] A flight field acquisition unit 605 is used to obtain a flight field for guiding the group by using thin plate spline interpolation of the final collective route, wherein the flight field is a large-scale dense deformation field;

[0170] A diffusion field acquisition unit 606, configured to obtain a diffusion field by using a route stop strategy in combination with the flight field estimation, wherein the diffusion field is a small-scale dense deformation field;

[0171] The migration field acquisition unit 607 is used to perform deformation field compounding of the flight field and the diffusion field to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to perform large-scale deformation compensation and small-scale deformation compensation on the floating image during the deformation process;

[0172] The cascade migration unit 608 is used to continuously deform the floating image through multiple migrations using the cascade migration framework and the migration field until a target deformed image is obtained after the migration converges.

[0173] The embodiment of the present application may also provide a large-scale deformation compensation device, the device comprising a processor and a memory:

[0174] The memory is used to store program code and transmit the program code to the processor;

[0175] The processor is used to execute the steps of the above-mentioned large-scale deformation compensation method according to the instructions in the program code.

[0176] like Figure 7 As shown, a large-scale deformation compensation device provided in an embodiment of the present application may include: a processor 10, a memory 11, a communication interface 12 and a communication bus 13. The processor 10, the memory 11 and the communication interface 12 communicate with each other through the communication bus 13.

[0177] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices, etc.

[0178] The processor 10 may call a program stored in the memory 11. Specifically, the processor 10 may execute operations in an embodiment of the large-scale deformation compensation method.

[0179] The memory 11 is used to store one or more programs, which may include program codes, and the program codes include computer operation instructions. In the embodiment of the present application, the memory 11 at least stores programs for implementing the following functions:

[0180] Extracting features from the fixed image and the floating image to obtain feature extraction results, and using a feature matching method and a parameter ratio test threshold in combination with the feature extraction results to identify and obtain a number of candidate routes, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image;

[0181] A route consistency quantification module is used to quantify the consistency of the candidate routes to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes;

[0182] Using an agglomerative clustering algorithm to determine a number of agglomerative routes from the consistency matrix;

[0183] A route checking strategy is used to determine a final collective route from a plurality of the clustered routes, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes;

[0184] Using thin plate spline interpolation of the final collective route to obtain a flight field for guiding the group, the flight field is a large-scale dense deformation field;

[0185] A diffusion field is obtained by using a route stop strategy in combination with the flight field estimation, wherein the diffusion field is a small-scale dense deformation field;

[0186] The flight field and the diffusion field are deformed to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to compensate for large-scale deformation and small-scale deformation of the floating image during the deformation process;

[0187] The floating image is deformed continuously through multiple migrations using a cascade migration framework and the migration field until a target deformed image is obtained after the migration converges.

[0188] In one possible implementation, the memory 11 may include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required for at least one function (such as a file creation function, a data reading and writing function), etc.; the data storage area can store data created during use, such as initialization data, etc.

[0189] In addition, the memory 11 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0190] The communication interface 12 may be an interface of a communication module, and is used to connect to other devices or systems.

[0191] Of course, it should be noted that Figure 7 The structure shown does not constitute a limitation on the large-scale deformation compensation device in the embodiment of the present application. In actual applications, the large-scale deformation compensation device may include Figure 7 More or fewer components than shown, or combinations of certain components.

[0192] The embodiment of the present application may also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store program codes, and the program codes are used to execute the steps of the above-mentioned large-scale deformation compensation method.

[0193] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0194] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0195] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0196] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A large-scale deformation compensation method, characterized in that: include: Extracting features from the fixed image and the floating image to obtain feature extraction results, and using a feature matching method and a parameter ratio test threshold in combination with the feature extraction results to identify and obtain a number of candidate routes, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image; A route consistency quantification module is used to quantify the consistency of the candidate routes to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes; Using an agglomerative clustering algorithm to determine a number of agglomerative routes from the consistency matrix; A route checking strategy is used to determine a final collective route from a plurality of the clustered routes, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes; Using thin plate spline interpolation of the final collective route to obtain a flight field for guiding the group, the flight field is a large-scale dense deformation field; A diffusion field is obtained by using a route stop strategy in combination with the flight field estimation, wherein the diffusion field is a small-scale dense deformation field; The flight field and the diffusion field are deformed to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to compensate for large-scale deformation and small-scale deformation of the floating image during the deformation process; The floating image is deformed continuously through multiple migrations using a cascade migration framework and the migration field until a target deformed image is obtained after the migration converges.

2. The large-scale deformation compensation method according to claim 1, characterized in that: All feature points with bijective matches are identified as candidate routes using the most relaxed ratio test.

3. The large-scale deformation compensation method according to claim 1, characterized in that: The distance consistency is expressed by the following formula: Where: k dis ∈(0, +∞] is used to control the sensitivity of the distance consistency d(i, j), d m Represents the relative distance in the floating image, d f Represents the relative distance in a fixed image; The Avalon-based voting method is used to quantify the topological consistency, which is expressed by the following formula: t(i, j) = t(i) t(j) Where: t(i) represents route v i The route-level topology of , t(j) represents the route v j The route level topology of route v j Indicates route v i Neighborhood routes.

4. The large-scale deformation compensation method according to claim 3, characterized in that: t(i) is expressed by the following formula: Where: k top ∈(0, +∞] is used to control the sensitivity of topology t(i), J rr (v i ) indicates v i Relative to its neighbors Topological changes; The J rr (v i ) is represented by the following formula: Where: J without (v i ) and J with (v i ) represent the introduction of v i Topology before and after.

5. The large-scale deformation compensation method according to claim 1, characterized in that: The final collective route is represented by the following formula: Where: c i Represents Ω i The contribution of the i-th collective route in Ω i represents the i-th region of interest.

6. The large-scale deformation compensation method according to claim 1, characterized in that: The route stop strategy collective route The drift distance is used as the penalty term in the objective function and is expressed as follows: Where: represents the normalized cross-correlation similarity, L rs Represents a route stop item, defined as k rs represents the parameter that weighs similarity and penalty, Indicates the flight field. represents the diffusion field.

7. The large-scale deformation compensation method according to claim 1, characterized in that: Before the cascade migration converges, the current collective route is merged into the previous collective route In the formula, it is expressed as follows: Where: Indicates the current collective route, Indicates the previous collective route.

8. A large-scale deformation compensation device, characterized in that: Used to perform the large-scale deformation compensation method according to any one of claims 1 to 7, the device comprises: A candidate route identification unit is used to extract features from a fixed image and a floating image to obtain feature extraction results, and to identify and obtain a number of candidate routes by combining the feature extraction results with a feature matching method and a parameter ratio test threshold, wherein the candidate routes are potential correspondences between feature points in the fixed image and feature points in the floating image; A consistency matrix acquisition unit, used to quantify the consistency of the candidate routes using a route consistency quantification module to obtain a consistency matrix; the route consistency quantification module is used to implement direction consistency quantification, distance consistency quantification and topology consistency quantification of the candidate routes; A clustering route determination unit, used to determine a number of clustering routes from the consistency matrix using a clustering algorithm; A final collective route determination unit is used to determine a final collective route from the plurality of clustered routes using a route checking strategy, wherein the final collective route is used to drive the image to perform large-scale deformation compensation; the route checking strategy is used to enhance the similarity of the local images so as to exclude negative contribution routes; A flight field acquisition unit, used for obtaining a flight field for guiding the group by using thin plate spline interpolation of the final collective route, wherein the flight field is a large-scale dense deformation field; A diffusion field acquisition unit, used to obtain a diffusion field by using a route stop strategy in combination with the flight field estimation, wherein the diffusion field is a small-scale dense deformation field; A migration field acquisition unit is used to perform deformation field compounding of the flight field and the diffusion field to obtain a migration field; the migration field is used to promote the deformation of the floating image, and to perform large-scale deformation compensation and small-scale deformation compensation on the floating image during the deformation process; The cascade migration unit is used to continuously deform the floating image through multiple migrations using the cascade migration framework and the migration field until a target deformed image is obtained after the migration converges.

9. A large-scale deformation compensation device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the large-scale deformation compensation method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the large-scale deformation compensation method described in any one of claims 1-7.

Images