Establishing key point matching model, key point matching, image registration method and device

Abstract

The embodiment of the present application discloses a method and device for establishing a key point matching model, key point matching, and image registration. The present application uses key point pairs corresponding to the same object position and different positions in two different modal images to respectively construct a second positive sample and a second negative sample, and adopts a contrastive learning method to train the second key point embedding submodel, and uses the trained second key point embedding submodel and matching submodel to construct a key point matching model, thereby realizing cross-modal image key point matching. And by adding disturbances to the image samples to obtain enhanced image samples, training the first key point embedding submodel of a single modality, and automatically obtaining the second training data after registering the cross-modal images, thereby reducing the cost of manual annotation. During the training process, the second key point embedding submodel is gradually and iteratively fine-tuned using the registered cross-modal images, thereby improving the model effect, and still being able to obtain good accuracy when there are large differences in the field of view.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method and device for establishing a key point matching model, key point matching, and image registration. Background Art

[0002] Image registration is the process of matching two or more images acquired at different times, with different imaging devices or under different conditions. Image registration is widely used in the fields of medical images, remote sensing images, traffic images, etc. Key point matching is a core technology in image registration, which is to determine whether the key point features in two images correspond to the same point by comparing them. If enough matching key points can be found in the two images, these key points can be used for image registration.

[0003] Traditional key point matching methods can achieve good results in key point matching between images of the same modality, but in many application scenarios, key point matching between cross-modal images is required. Therefore, an accurate cross-modal key point matching method is urgently needed. Summary of the invention

[0004] In view of this, the present application provides a method and device for establishing a key point matching model, key point matching, and image registration, so as to accurately achieve cross-modal key point matching.

[0005] This application provides the following solutions:

[0006] In a first aspect, a method for establishing a key point matching model is provided, the method comprising:

[0007] Acquire second training data including a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to positions of the same object collected from the first modality image and the second modality image, respectively, and the second negative sample pairs include key points corresponding to positions of different objects collected from the first modality image and the second modality image, respectively, and the first modality image and the second modality image are collected for the same object;

[0008] The second key point embedding submodel is trained using the second training data, wherein the second key point embedding submodel performs feature extraction of one or more scales on each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair;

[0009] A key point matching model is obtained using the trained second key point embedding submodel and the matching submodel, wherein the matching submodel is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding submodel.

[0010] According to an achievable manner in the embodiment of the present application, before obtaining the second training data including a plurality of positive sample pairs and negative sample pairs, the method further includes:

[0011] Acquire first training data including a plurality of first positive sample pairs and first negative sample pairs, wherein the first positive sample pairs include key points corresponding to the same object position collected from the first image sample and the second image sample, respectively, and the first negative sample pairs include key points corresponding to different object positions collected from the first image sample and the second image sample, respectively; wherein the first image sample and the second image sample are images obtained by adding different disturbances to the same image sample, or the first image sample and the second image sample are an image sample and an image obtained by adding disturbances to the image sample, respectively;

[0012] The first training data is used to train a first key point embedding submodel, wherein the first key point embedding submodel performs feature extraction of one or more scales on each key point in the input first image sample and each key point in the second image sample to obtain feature representations of one or more scales corresponding to each key point in the first image sample and feature representations of one or more scales corresponding to each key point in the second image sample; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the first positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the first negative sample pair;

[0013] The initial first modality image and the second modality image are registered using the trained first key point embedding sub-model, and the second training data is obtained using the registered first modality image and the second modality image.

[0014] According to an achievable manner in an embodiment of the present application, after the second key point embedding sub-model is trained using the second training data, the method further includes: registering the first modality image and the second modality image using the second key point embedding sub-model obtained by the current training, and updating the second training data using the registered first modality image and the second modality image; and continuing to execute the step of training the second key point embedding sub-model using the second training data until a preset iteration stop condition is met;

[0015] The method of obtaining a key point matching model by using the second key point embedding sub-model and the matching sub-model obtained through training includes: obtaining a key point matching model by using the second key point embedding sub-model and the matching sub-model obtained after satisfying a preset iteration stop condition.

[0016] According to an achievable method in an embodiment of the present application, the registration includes:

[0017] Determine a plurality of matching key point pairs in the first modality image and the second modality image using feature representations of one or more scales corresponding to each key point in the first modality image and the second modality image;

[0018] Determining a transformation matrix between the first modality image and the second modality image using the plurality of matching key point pairs;

[0019] The second modality image is transformed using the transformation matrix to obtain the second modality image registered with the first modality image.

[0020] According to an achievable method in an embodiment of the present application, adding disturbance to the image sample includes: performing at least one of super-pixel affine transformation, adding noise, intensity transformation and blurring processing on the image sample.

[0021] In a second aspect, a cross-modal key point matching method is provided, the method comprising:

[0022] Acquire a first modality image and a second modality image to be matched, where the first modality image and the second modality image are acquired for the same type of object;

[0023] Inputting the first modality image and the second modality image to be matched into a key point matching model pre-established by the method described in the first aspect above, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model;

[0024] The second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be matched respectively;

[0025] The matching sub-model uses feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be matched, to match the key points in the first modality image and the second modality image to be matched.

[0026] In a third aspect, a cross-modal image registration method is provided, the method comprising:

[0027] Acquire a first modality image and a second modality image to be registered, wherein the first modality image and the second modality image are acquired for the same type of object;

[0028] Inputting the first modality image and the second modality image to be registered into a key point matching model pre-established by the method described in the first aspect above, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model;

[0029] The second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively;

[0030] The matching sub-model uses feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered, to match key points in the first modality image and the second modality image to be registered;

[0031] The first modality image and the second modality image to be registered are registered using the multiple matching key point pairs obtained by the matching.

[0032] According to an achievable manner in an embodiment of the present application, the method is applied to a medical scenario, and the first modality image and the second modality image to be registered include: a first modality medical image and a second modality medical image acquired for the same type of animal body part.

[0033] In a fourth aspect, a cross-modal image registration method is provided, the method comprising:

[0034] Displaying the first modality image and the second modality image to be registered on a display screen, a virtual reality VR device, or an augmented display AR device, where the first modality image and the second modality image are acquired for the same type of object;

[0035] Inputting the first modality image and the second modality image to be registered into a key point matching model pre-established by the method described in the first aspect above, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model;

[0036] The second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively;

[0037] The matching sub-model uses feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered, to match key points in the first modality image and the second modality image to be registered;

[0038] Using the multiple matching key point pairs obtained by the matching, registering the first modality image and the second modality image to be registered;

[0039] The display screen, VR device or AR device is driven to render and display the registered first modality image and second modality image.

[0040] In a fifth aspect, a device for establishing a key point matching model is provided, the device comprising:

[0041] a second sample acquisition unit, configured to acquire second training data comprising a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to positions of the same object collected from the first modality image and the second modality image, respectively; and the second negative sample pairs include key points corresponding to positions of different objects collected from the first modality image and the second modality image, respectively; and the first modality image and the second modality image are collected for the same object;

[0042] The second model training unit is configured to train a second key point embedding submodel using the second training data, wherein the second key point embedding submodel performs feature extraction of one or more scales on each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair;

[0043] The model building unit is configured to obtain a key point matching model using the second key point embedding sub-model and the matching sub-model trained by the second model training unit, wherein the matching sub-model is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding sub-model.

[0044] In a sixth aspect, a cross-modal key point matching device is provided, the device comprising:

[0045] An image acquisition unit is configured to acquire a first modality image and a second modality image to be matched, wherein the first modality image and the second modality image are acquired for the same type of object;

[0046] A key point matching unit is configured to input the first modality image and the second modality image to be matched into a key point matching model pre-established by the device as described in the fifth aspect above, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be matched respectively; the matching sub-model matches the key points in the first modality image and the second modality image to be matched using the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be matched respectively.

[0047] In a seventh aspect, a cross-modal image registration device is provided, the device comprising:

[0048] An image acquisition unit is configured to acquire a first modality image and a second modality image to be registered, wherein the first modality image and the second modality image are acquired for the same type of object;

[0049] A key point matching unit is configured to input the first modality image and the second modality image to be registered into a key point matching model pre-established by the device as described in the fifth aspect above, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively; the matching sub-model uses the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered respectively, to match the key points in the first modality image and the second modality image to be registered;

[0050] The image registration unit is configured to register the first modality image and the second modality image to be registered using the multiple matching key point pairs obtained by the matching.

[0051] According to an eighth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the first to fourth aspects are implemented.

[0052] According to a ninth aspect, there is provided an electronic device, comprising:

[0053] one or more processors; and

[0054] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first to fourth aspects above.

[0055] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0056] 1) This application constructs a second positive sample from key point pairs corresponding to the same object position in two different modal images, and constructs a second negative sample from key point pairs corresponding to different object positions, and trains the second key point embedding submodel by contrastive learning, so that the second key point embedding submodel can learn the feature representation of more than one scale of each key point in different modal images. The key point matching model is constructed using the trained second key point embedding submodel and matching submodel, thereby achieving cross-modal image key point matching.

[0057] 2) The present application obtains the first image sample and the second image sample by adding different disturbances to the image sample, or uses the image sample and the image obtained by adding disturbances to the image sample as the first image sample and the second image sample respectively. Then, the first training data is constructed using the first image sample and the second image sample, and the first key point embedding submodel applied to a single modality is trained by contrastive learning, the initial first modality image and the second modality image are aligned using the first key point embedding submodel, and the second training data is obtained using the aligned first modality image and the second modality image. This method can automatically generate the second training data used to train the cross-modal second key point embedding submodel, greatly reducing labor costs, and can also obtain good cross-modal matching performance without manual annotation.

[0058] 3) The present application adds perturbations to the image samples so that the obtained first image samples and second image samples retain the topological structure of the objects in the image samples, and the visual perturbations also improve the robustness of the trained first key point embedding sub-model.

[0059] 4) This application first trains the second key point embedding sub-model by contrastive learning, then uses the trained second key point embedding sub-model to align the cross-modal data, updates the second training data, and further iterates the training of the second key point embedding sub-model. This gradual fine-tuning method improves the effect of the second key point embedding sub-model, and can still achieve good accuracy when there are large differences in the field of view.

[0060] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 is a system architecture diagram applicable to the embodiments of the present application;

[0063] Figure 2 A flow chart of a method for establishing a key point matching model provided in an embodiment of the present application;

[0064] Figure 3 A flowchart of a preferred method for establishing a key point matching model provided in an embodiment of the present application;

[0065] Figure 4 A schematic structural diagram of the first key point embedding sub-model provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the key point matching model provided in the embodiment of the present application;

[0067] Figure 6 A flowchart of a cross-modal key point matching method provided in an embodiment of the present application;

[0068] Figure 7 A flowchart of a cross-modal image registration method provided in an embodiment of the present application;

[0069] Figure 8 A flow chart of an image registration method applied to medical scenarios provided in an embodiment of the present application;

[0070] Figure 9a Examples of CT images and MRI images provided in the embodiments of the present application;

[0071] Figure 9b An example diagram of a registered CT image and MRI image provided in an embodiment of the present application;

[0072] Fig.9c An example diagram of matching key points of a CT image and an MRI image provided in an embodiment of the present application;

[0073] Fig.10 A schematic block diagram of a device for establishing a key point matching model provided in an embodiment of the present application;

[0074] Fig.11 A schematic block diagram of a cross-modal key point matching device provided in an embodiment of the present application;

[0075] Fig.12 A schematic block diagram of a cross-modal image registration device provided in an embodiment of the present application;

[0076] Fig.13 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0078] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0079] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0080] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0081] In order to facilitate the understanding of the present application, the system architecture to which the present application is applicable is first briefly described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , the system architecture includes a model training device, a key point matching device and an image registration device on the server side, and may also include a user terminal.

[0082] The model training device is used to perform model training in an offline stage. That is, after obtaining the training data, the method provided in the embodiment of the present application can be used to perform model training to obtain a key point matching model.

[0083] As one of the achievable methods, the key point matching device is used to use the trained key point matching model online to perform key point matching on the first modality image and the second modality image to be matched, and obtain matching key point pairs. Each matching key point pair includes two key points, a first key point and a second key point, wherein the first key point and the second key point are respectively located in the first modality image and the second modality image, and are determined to correspond to the same object position.

[0084] As another feasible method, the first modality image and the second modality image to be registered are firstly submitted to a key point matching device for key point matching to obtain a matching key point pair; and then the image registration device uses the matching key point pair to register the first modality image and the second modality image to be registered. The so-called registration is to fix one of the images (usually called the fixed image) and transform the other image (usually called the moving image) (including rotation, translation, etc.) so that the transformed moving image and the overlapping area in the fixed image are aligned as much as possible.

[0085] The model training device, key point matching device and image registration device can be set as independent servers, or can be set on the same server or server group, or can be set on an independent or the same cloud server. Cloud server, also known as cloud computing server or cloud host, is a host product in the cloud computing service system to solve the defects of difficult management and weak service scalability in traditional physical host and virtual private server (VPS) services. The model training device, key point matching device and image registration device can also be set on a computer terminal with strong computing power.

[0086] As one of the achievable methods, a user can send the first modality image and the second modality image to be matched to the key point matching device of the server through the network through the user terminal. After the key point matching device uses the method provided in the embodiment of the present application to perform key point matching on the first modality image and the second modality image to be matched, the matching result, i.e., the information of the matched key point pair, is returned to the user terminal.

[0087] As another achievable method, the user can send the first modality image and the second modality image to be registered to the image registration device on the server through the network through the user terminal. After the image registration device uses the method provided in the embodiment of the present application to register the first modality image and the second modality image to be registered, the registration result is returned to the user terminal.

[0088] The above-mentioned user terminals may include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, smart medical devices, PCs (Personal Computers), etc. Among them, smart mobile devices may include such as mobile phones, tablet computers, laptops, PDAs (Personal Digital Assistants), Internet cars, etc. Smart home devices may include such as smart TVs, smart refrigerators, etc. Wearable devices may include such as smart watches, smart glasses, smart bracelets, etc. The above-mentioned user terminals have a display screen that can display key point matching results or image registration results. The above-mentioned wearable devices may also include: VR (Virtual Reality) devices, AR (Augmented Reality), mixed reality devices (i.e., devices that can support virtual reality and augmented reality), etc., which display key point matching results or image registration results through VR, AR or mixed reality technology.

[0089] It should be noted that the key point matching device and image registration device can perform key point matching and image registration in an offline manner in addition to performing key point matching and image registration online, for example, performing key point matching and image registration on batches of image pairs respectively.

[0090] It should be understood that Figure 1 The number of model training devices, key point matching devices, image registration devices, key point matching models and user terminals in the embodiment is only for illustration. According to the implementation requirements, any number of model training devices, key point matching devices, image registration devices, key point matching models and user terminals may be provided.

[0091] Figure 2 A flow chart of a method for establishing a key point matching model provided in an embodiment of the present application. The method can be performed by Figure 2 The model training device in the system shown is executed. Figure 2 As shown in , the method may include the following steps:

[0092] Step 202: Obtain second training data including a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to the same object position collected from the first modality image and the second modality image respectively, and the second negative sample pairs include key points corresponding to different object positions collected from the first modality image and the second modality image respectively, and the first modality image and the second modality image are collected for the same object.

[0093] Step 204: Use the second training data to train the second key point embedding submodel, wherein the second key point embedding submodel performs feature extraction of one or more scales for each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair.

[0094] Step 206: A key point matching model is obtained using the trained second key point embedding sub-model and the matching sub-model, wherein the matching sub-model is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding sub-model.

[0095] As can be seen from the above process, the present application constructs a second positive sample from key point pairs corresponding to the same object position in two different modal images, and constructs a second negative sample from key point pairs corresponding to different object positions, and trains the second key point embedding submodel by contrastive learning, so that the second key point embedding submodel can learn the feature representation of more than one scale of each key point in different modal images. The key point matching model is constructed using the trained second key point embedding submodel and matching submodel, thereby achieving cross-modal image key point matching.

[0096] It should be noted that the limitations such as "first" and "second" involved in the present disclosure do not have limitations in terms of size, order, quantity, etc., and are only used to distinguish them in name. For example, "first training data" and "second training data" are used to distinguish two sets of training data in name. Another example is "first modality image" and "second modality image" are used to distinguish two modal images in name. Another example is "first key point embedding submodel" and "second key point embedding submodel" are used to distinguish two key point embedding submodels in name. And so on.

[0097] The above method is described in detail below in conjunction with a preferred embodiment. Figure 3 A flowchart of a preferred method for establishing a key point matching model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown in , the method may include the following steps:

[0098] Step 302: Obtain first training data including a plurality of first positive sample pairs and first negative sample pairs, wherein the first positive sample pair includes key points corresponding to the same object position collected from the first image sample and the second image sample respectively, and the first negative sample pair includes key points corresponding to different object positions collected from the first image sample and the second image sample respectively, and the first image sample and the second image sample are images obtained by adding different disturbances to the same image sample, or the first image sample and the second image sample are an image sample and an image obtained by adding disturbances to the image sample, respectively.

[0099] Steps 302 to 306 in this process are Figure 2 The process shown is previously executed in order to automatically obtain second training data using the unimodal first key point embedding sub-model for training the cross-modal second key point embedding sub-model.

[0100] The first key point embedding submodel of the unimodal can output the feature representation of each element (Token) in the image when a unimodal image is input. The goal is to ensure that the feature representations corresponding to the same position or similar position of objects in different images are the same or similar. Each Token of an image refers to the elements that constitute the image. For an image, the image is divided into a sequence of non-overlapping blocks, and the blocks in the image and the start symbols of the sequence are all Tokens. For a two-dimensional image, the blocks therein may be composed of one or more pixels. For a three-dimensional image, the blocks therein may be composed of one or more voxels. The key points involved in the embodiments of the present application are usually points with specific features of the objects contained in the image, and the positions and topologies are easy to mark the objects, and have scale invariance, that is, they can represent the same object in a same or highly similar invariant form in similar images containing the same scene or target. It can be understood that each Token of the image contains key points.

[0101] Since the first key point embedding submodel is unimodal, the requirements for the embedding effect are relatively low compared to the cross-modal second key point embedding submodel. Therefore, the first training data used to train the first key point embedding submodel in this step can be derived from image samples of the same modality. As one of the feasible ways, after obtaining the image sample, different perturbations can be added to the same image sample to obtain a first image sample and a second image sample. In this way, the position of the key point in the image sample can be recorded, and the two key point positions corresponding to the key point after being processed with different perturbations are recorded. These two key point positions constitute a first positive sample pair. Randomly sampling two key points in the first image sample and the second image sample constitutes a first negative sample pair.

[0102] As another feasible method, the image sample and the image obtained after adding disturbance to the image sample can be used as the first image sample and the second image sample respectively. In this way, the key point can be recorded in the image sample, and then the position of the key point after adding disturbance, that is, the key point in the second image sample, is recorded, thereby forming a key point pair as the first positive sample pair. The key point is recorded in the image sample, and then a key point is randomly sampled in the second image sample, thereby forming a key point pair as the first negative sample pair.

[0103] Then, key points are sampled from the first image sample and the second image sample respectively to construct a positive sample pair and a negative sample pair, which are referred to as a first positive sample pair and a first negative sample pair.

[0104] The perturbation added to the image sample may include at least one of affine transformation, noise addition, intensity transformation, blurring, etc. The added perturbation may be random, and the generated first image sample and second image sample retain the topological structure of the object in the original image sample, and the visual perturbation improves the robustness of the trained first key point embedding sub-model.

[0105] In order to further improve the effect of training the first key point embedding sub-model, the image samples can be segmented to obtain each superpixel. Then, the perturbation is added at the superpixel granularity, that is, the same perturbation is added to the same superpixel, and different perturbations can be added to different superpixels. The so-called superpixel refers to an area where perceptually similar pixels are grouped together, which is an area with more compact features. The above segmentation process can be implemented by methods such as SLIC (Simple Linear Iterative Clustering).

[0106] Step 304: Use the first training data to train the first key point embedding submodel, wherein the first key point embedding submodel performs feature extraction of one or more scales for each key point in the input first image sample and each key point in the second image sample to obtain feature representations of one or more scales corresponding to each key point in the first image sample and feature representations of one or more scales corresponding to each key point in the second image sample; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the first positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the first negative sample pair.

[0107] The first key point embedding submodel provided in the embodiment of the present application can extract feature representations of more than one scale corresponding to each Token (including each feature point) from the input image (the first image sample or the second image sample in this step). Among them, the first key point embedding submodel mainly includes a feature extraction network and a fusion network. The feature extraction network is used to perform multi-layer feature extraction on the input image. The feature extraction network can be implemented by a convolutional neural network or by a Transformer-based network. The fusion network is used to obtain feature representations of more than one scale for each key point in the image sample using the multi-layer features extracted by the feature extraction network. The fusion network can adopt a method such as FPN (Feature Pyramid Networks). Here, the feature representation of two scales is described as an example.

[0108] As one possible way to implement this, the structure of the first keypoint embedding sub-model is as follows: Figure 4 As shown in the figure, the multi-layer convolution network on the left corresponds to the feature extraction network, and the multi-layer convolution network on the right corresponds to the fusion network. The input image can be downsampled through the multi-layer convolution network, and the result of the last downsampling layer is convolved and normalized (for example, L2 normalization) to obtain the first scale feature representation. Figure 4 In the above diagram, it is represented as global embedding. The sampling results of other layers are upsampled in multiple layers. Each layer uses the downsampling result corresponding to this layer to fuse the upsampling result of the previous layer and then upsamples to obtain the upsampling result of this layer. Then, the upsampling result of the last layer is convolved and normalized (for example, L2 normalization) to obtain the feature representation of the second scale. Figure 4 It is represented as Local Embedding in .

[0109] When training the first keypoint embedding submodel, for a keypoint p in an image sample (denoted as X) and a keypoint p' in a second image sample (denoted as X') in a sample pair, the first keypoint embedding submodel obtains a global embedding F for p g and local embedding F l , we get the global embedding F for p' g' and local embedding F l' . Use self-supervised learning methods, such as contrastive learning, to try to get closer to F in the first positive sample pair g With F g' , and F l With F l' The distance between them is as far as possible, and the distance between the first negative sample and F is as far as possible. g With F g' , and F l With F l' The distance between.

[0110] In the embodiments of the present specification, a loss function can be constructed according to the above training objectives, and the value of the loss function can be used in each round of iteration to update the model parameters by methods such as gradient descent until the preset training end condition is met. The training end condition may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0111] Step 306: Use the trained first key point embedding sub-model to register the initial first modality image and the second modality image, and use the registered first modality image and the second modality image to obtain second training data.

[0112] The first modality image and the second modality image are cross-modal images. The so-called cross-modal images refer to images acquired by different imaging devices. For example, images acquired by a camera and an infrared device belong to different modalities. For example, images acquired by CT (Computed Tomography) and MR (Magnetic Resonance) belong to different modalities, and so on.

[0113] After obtaining the first key point embedding submodel, the first key point embedding submodel can be used to obtain feature representations of one or more scales corresponding to each key point in the initial first modality image, and feature representations of one or more scales corresponding to each key point in the initial second modality image. By calculating the distance between the feature representations of each key point in the first modality image and each key point in the second modality image, multiple matching key point pairs in the first modality image and the second modality image are determined. Each matching key point pair can be represented as p i and p i ', i = 1, ..., N, N is the number of matching key point pairs. i and p i ' respectively come from the first modality image and the second modality image, p i ' can be the second modality image with p i ' is the key point with the highest feature representation similarity. When calculating the feature representation distance between two key points, the feature representations of more than one scale corresponding to each key point can be integrated first, and then the similarity between the integrated feature representations of the two key points can be calculated. The similarity between the feature representations of each scale corresponding to the two key points can also be calculated separately, and then the similarities corresponding to each scale are integrated to obtain the feature representation similarity of the two key points.

[0114] Then, using multiple matching key point pairs, the transformation matrix between the first modality image and the second modality image is determined. i '=Rpi +T+N i , R is the rotation matrix, T is the translation vector, N i is the instance noise. Here we can understand that R and T are calculated to minimize ∑ 2 :

[0115]

[0116] The second modality image is then transformed using the transformation matrix (i.e., the above R and T) to obtain the second modality image registered with the first modality image. The registered first modality image and the second modality image are used to construct second training data to train the second key point embedding sub-model.

[0117] The above process is actually a process of pre-training a single-modal first key point embedding sub-model and automatically obtaining the second training data using the single-modal first key point embedding sub-model. The above process can automatically generate the second training data without manually annotating the second positive sample pairs and the second negative sample pairs of the first modality images and the second modality images of the two modalities, which greatly saves labor costs.

[0118] In addition to the above steps 302 to 306, other implementation methods may also be used, such as using other methods such as VIT (Vision Transformer), RAN (Residual Attention Network), U-Net (U-Net is a variant of FCN (Fully Convolutional Network), which was proposed to solve the problem of biomedical imaging. Due to its good effect, it has been widely used in various fields of image segmentation) as the first key point embedding sub-model. For another example, the first modality image and the second modality image that are aligned as much as possible may be used as the second training data, and so on.

[0119] Step 308: Obtain second training data including a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to the same object position collected from the first modality image and the second modality image respectively, and the second negative sample pairs include key points corresponding to different object positions collected from the first modality image and the second modality image respectively, and the first modality image and the second modality image are collected for the same object.

[0120] This step and Figure 2 This corresponds to step 202 in the illustrated process.

[0121] As one of the feasible ways, a pair of key points corresponding to the same object position can be selected from the overlapping area of ​​the first modality image and the second modality image obtained after registration as the second positive sample pair, which is expressed as The key point pairs corresponding to different object positions are used as the second negative sample pairs, expressed as In addition, key point pairs can be selected from the non-overlapping areas of the first modality image and the second modality image obtained after registration to supplement the second negative sample pair, which is expressed as It can help the second key point embedding sub-model learn the features of images under different fields of view, so that when the second key point embedding sub-model performs feature extraction of cross-modal images, it can also achieve better results for images under different fields of view. The so-called field of view (FOV) refers to the range of objects captured by the acquisition device at a certain distance.

[0122] Step 310: Use the second training data to train the second key point embedding submodel, wherein the second key point embedding submodel performs feature extraction of one or more scales for each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair.

[0123] This step and Figure 2 This corresponds to step 204 in the illustrated process.

[0124] The model structure of the second key point embedding sub-model may be the same as or different from the model structure of the first key point embedding sub-model. However, through the training in this step, the second key point embedding sub-model has the cross-modal feature extraction capability.

[0125] In the embodiment of the present application, a loss function can be constructed according to the above training objectives, and the value of the loss function is used in each round of iteration to update the model parameters by methods such as gradient descent until the preset training end condition is met. The training end condition may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0126] As one of the possible ways to implement it, the loss function L can be formulated as follows:

[0127]

[0128] Among them, x iand x i 'are the two key points in the second positive sample pair, x j For x i At the corresponding key point in the second negative sample pair, τ is the temperature coefficient.

[0129] Step 312: Determine whether a preset iteration stop condition is met, if so, execute step 316; otherwise, execute step 314.

[0130] Step 314: Use the second key point embedding sub-model obtained by the current training to align the first modality image and the second modality image, use the aligned first modality image and the second modality image to update the second training data, and go to step 310.

[0131] The registration process in this step is similar to step 306, using the feature representation of one or more scales corresponding to each key point in the first modality image and the second modality image, multiple matching key point pairs in the first modality image and the second modality image are determined; using the multiple matching key point pairs, the transformation matrix between the first modality image and the second modality image is determined; using the transformation matrix, the second modality image is transformed to obtain the second modality image registered with the first modality image. This will not be described in detail here.

[0132] The second key point embedding sub-model is iteratively trained using the second training data updated after registration until a preset iteration stop condition is met. The iteration stop condition may be, for example, the number of loop iterations of the second key point embedding sub-model reaches a preset number of times, the accuracy of key point matching on the test set using the second key point embedding sub-model reaches a preset accuracy threshold, and so on.

[0133] Through the above-mentioned alignment and iterative training process of the second key point embedding sub-model, a more precise fine-tuning process is actually achieved. The second key point embedding sub-model usually converges after several iterations. The converged model is used as the final second key point embedding sub-model to construct the key point matching model, that is, execute step 316.

[0134] Step 316: Obtain a key point matching model using the trained second key point embedding sub-model and the matching sub-model, wherein the matching sub-model is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding sub-model.

[0135] This step and Figure 2 This corresponds to step 206 in the illustrated flow.

[0136] like Figure 5As shown in , the key point matching model may include a second key point embedding sub-model and a matching sub-model, which can be used to perform key point matching on cross-modal images. The cross-modal key point matching method is described in detail below in conjunction with the embodiments. Figure 6 As shown, the cross-modal key point matching method may include the following steps:

[0137] Step 602: Acquire a first modality image and a second modality image to be matched, where the first modality image and the second modality image are acquired for the same type of object.

[0138] For the same type of object, images acquired by different imaging devices are used as the first modality image and the second modality image. For example, images acquired by a camera and an infrared device belong to different modalities. For example, images acquired by CT (Computed Tomography) and MR (Magnetic Resonance) belong to different modalities, and so on.

[0139] For example, the first modality image and the second modality image are both images of different modalities acquired from human chest organs, but they may be of the same person or different persons. For another example, the first modality image and the second modality image are both images of different modalities acquired from human abdominal organs, but they may be of the same person or different persons.

[0140] Step 604: input the first modality image and the second modality image to be matched into a pre-established key point matching model, the key point matching model including a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be matched respectively; the matching sub-model matches the key points in the first modality image and the second modality image to be matched using the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be matched respectively.

[0141] The first modality image is denoted as X1, and the second modality image is denoted as X2, such as Figure 5 As shown in , the second key point embedding submodel in the key point matching model extracts the feature representation of each Token from X1 and X2 respectively, where each Token includes each key point. Each key point can be pre-specified, or can be detected by any key point detection technology, which is not limited in this application. In other words, the second key point embedding submodel can obtain the feature representation of more than one scale corresponding to each key point in X1 and X2.

[0142] Then the matching sub-model uses the feature representation of one or more scales corresponding to each key point in X1 and X2 to match the key points. 1i , find the key point x in X2 1i The feature between represents the key point x that is closest to 1j Construct a matching key point pair to find the matching key point pair between X1 and X2.

[0143] When calculating the distance of the feature representation between two key points, the feature representations of more than one scale corresponding to each key point can be integrated to obtain the integrated feature representation corresponding to each key point, and then the similarity between the integrated feature representations of the two key points can be calculated. The similarity between the feature representations of each scale corresponding to the two key points can also be calculated, and then the similarities corresponding to each scale can be integrated to obtain the similarity of the feature representations of the two key points.

[0144] In the embodiments of the present application, when calculating the distance between feature representations, methods such as Euclidean distance, Manhattan distance, Chebyshev distance, etc. can be used. When matching, simple sorting methods, NN (Nearest Neighbor) search methods, etc. can be used. The present application does not impose any restrictions on this.

[0145] In some scenarios, it is necessary to register cross-modal images, that is, to align the regions corresponding to the same object position in the cross-modal images as much as possible. Figure 7 The process shown in Figure 7 As shown in , the cross-modal image registration method can include the following process:

[0146] Step 702: Acquire a first modality image and a second modality image to be registered, wherein the first modality image and the second modality image are acquired for the same type of object.

[0147] Step 704: input the first modality image and the second modality image to be registered into a pre-established key point matching model, the key point matching model including a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively; the matching sub-model matches the key points in the first modality image and the second modality image to be registered using the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered respectively.

[0148] The above steps 702 to 704 are Figure 6 Steps 602 to 604 are similar and will not be described in detail here.

[0149] Step 706: using the multiple matching key point pairs obtained by matching, register the first modality image to be registered and the second modality image.

[0150] The registration process in this step is similar to step 306, where a transformation matrix between the first modality image and the second modality image is determined using multiple matching key point pairs; the second modality image is transformed using the transformation matrix to obtain the second modality image registered with the first modality image. This will not be described in detail here.

[0151] As one of the feasible embodiments, the above-mentioned image segmentation method can be applied to ordinary display screens, and can also be applied to VR (Virtual Reality) or AR (Augmented Reality) scenarios.

[0152] First, the first modality image and the second modality image to be registered can be displayed on the display screen, VR or AR device; the first modality image and the second modality image to be registered are input into the key point matching model to obtain multiple matching key point pairs of the first modality image and the second modality image; then the first modality image and the second modality image to be registered are registered using the multiple matching key point pairs obtained by matching; finally, the display screen, VR device or AR device is driven to render and display the registered first modality image and the second modality image. In addition, the correspondence between the matching key point pairs can also be displayed on the display screen, VR device or AR device. For example, when a user selects a key point on the first modality image, the key point matching the selected key point can be highlighted on the second modality image. The highlighting method can be, for example, highlighting, special color, etc.

[0153] Through the content of the above embodiments, the user can view the registration results of the two modal images and the key point matching results through a display screen, a VR device or an AR device. Among them, this application does not limit how to map the first modality image, the second modality image, the key points, etc. into the virtual space structure, and any method in the prior art can be used.

[0154] The image registration method provided in the above embodiments of the present application can be applied to a variety of scenarios. For example, in a criminal investigation scenario, images containing multiple faces can be registered to help criminal investigators locate the same face in different images and compare them. For another example, image registration can be performed in an image stitching scenario to find overlapping areas to facilitate image stitching, etc.

[0155] It can also be applied to other scenarios, but one of the more important scenarios is the medical scenario, where the method provided in the embodiment of the present application can be used to register cross-modal medical images to facilitate doctors (such as radiologists) to compare medical images of different modalities. This application scenario is described below as an example.

[0156] Figure 8 The flowchart of the image registration method applied to medical scenes provided in the embodiment of the present application is as follows: Figure 8 As shown in , the method may specifically include the following steps:

[0157] Step 802: Acquire a first modality medical image and a second modality medical image acquired for the same type of human body part.

[0158] The medical images may be modalities such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), and ultrasound examination images.

[0159] The so-called first modality medical image and second modality medical image collected for the same type of human body part means that the first modality medical image and the second modality medical image at least contain the same type of human body part, but there may be differences in field of view, angle, etc. In addition to the human body, it can also be extended to other animal bodies.

[0160] For example, CT images and MRI images are collected for the head and neck of the same user. Since the CT images and MRI images of the two modes for the same user's head and neck may have deviations in shooting angle, shooting distance, etc., and the two modes themselves have different imaging results due to different acquisition methods, it is hoped that the collected CT images and MR images can be registered.

[0161] Step 804: input the first modality medical image and the second modality medical image to be registered into a pre-established key point matching model, and use the key point matching model to match the key points of the first modality medical image and the second modality medical image.

[0162] Step 806: using the multiple matching key point pairs obtained by matching, register the first modality medical image and the second modality medical image.

[0163] For the key point matching and registration process, please refer to the relevant records in the previous embodiments, which will not be repeated here.

[0164] For example, Figure 9aAs shown in , CT images and MRI images of the head and neck of the same user are taken as the two modal images to be registered. After the registration is implemented in the above manner, the regions corresponding to the same part can be aligned, such as Figure 9b The key point pairs that are matched can be schematically represented as Fig.9c As shown in . The matching results and matching results can be intuitively displayed to the physician, or provided to the downstream device for further processing such as image segmentation.

[0165] The above method was experimented on a dataset of CT images and MRI images collected for the head and neck. The dataset consists of 120 pairs of CT images and MRI images. MRI mainly collects the area between the nose and the second cervical vertebra, and CT images include the area from the top of the head to the middle of the lungs. 100 image pairs were used for training and 20 image pairs were used for testing. The difference between the position of the key point on the CT image corresponding to the key point on the MRI image obtained after the registration process and the true value (that is, the true position of the key point on the CT image corresponding to the key point on the MRI image) was expressed by MED (Mean Euclidean Distance). The MEDs on the X-axis, Y-axis and Z-axis are 0.87±0.83 (in voxels), 1.89±1.66 and 2.29±1.74 respectively, and the overall MED is 3.14±2.11. It can be seen that the above method has achieved very good registration results on the dataset of CT images and MRI images collected for the head and neck.

[0166] The above method is experimented on a data set of CT images and MRI images collected for the abdomen. When organs such as the intestines and stomach in the abdomen are scanned, the shape will change significantly, which will affect the position of other organs, so that the traditional registration cannot align the organs well. And the nuclear magnetic resonance scan may only capture part of the organ, which increases the difficulty of registration between the CT image and the MRI image, causing the traditional registration method to be unable to handle this situation, resulting in poor registration effect. The above method provided by the embodiment of the present application is used to experiment on a data set, and the data set consists of 98 pairs of CT images and MRI images. MRI mainly collects a small field of view around the liver and kidneys, and the CT image includes the area from the chest to the bottom of the pelvis. Using 80 image pairs for training and 18 image pairs for testing, the MEDs on the X-axis, Y-axis and Z-axis are 3.63±2.86 (in voxels), 6.16±5.69 and 8.70±6.40, respectively, and the overall MED is 12.86±6.54. It can be seen that the above method can still achieve very good registration effect on the data set of CT images and MRI images collected for the abdomen.

[0167] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] According to an embodiment of another aspect, a device for establishing a key point matching model is provided. Fig.10 A schematic block diagram of a device for establishing a key point matching model according to an embodiment is shown, and the device corresponds to Figure 1 The model training device in the system shown. Fig.10 As shown, the device 1000 includes: a second sample acquisition unit 1001, a second model training unit 1002 and a model construction unit 1003, and may further include a first sample acquisition unit 1004, a first model training unit 1005 and a registration processing unit 1006. The main functions of each component unit are as follows:

[0169] The second sample acquisition unit 1001 is configured to acquire second training data including a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to the same object position collected from the first modality image and the second modality image respectively, and the second negative sample pairs include key points corresponding to different object positions collected from the first modality image and the second modality image respectively, and the first modality image and the second modality image are collected for the same object.

[0170] The second model training unit 1002 is configured to train a second key point embedding sub-model using the second training data, wherein the second key point embedding sub-model performs feature extraction of one or more scales on each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair.

[0171] The model building unit 1003 is configured to obtain a key point matching model using the second key point embedding sub-model and the matching sub-model obtained by training the second model training unit, wherein the matching sub-model is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding sub-model.

[0172] Furthermore, the first sample acquisition unit 1004 can be configured to acquire first training data comprising multiple first positive sample pairs and first negative sample pairs, the first positive sample pair comprising key points corresponding to the same object position collected from the first image sample and the second image sample respectively, and the first negative sample pair comprising key points corresponding to different object positions collected from the first image sample and the second image sample respectively; wherein the first image sample and the second image sample are images obtained by adding different disturbances to the same image sample, or the first image sample and the second image sample are an image sample and an image obtained by adding disturbances to the image sample, respectively.

[0173] The first sample acquisition unit 1004 may perform at least one of super-pixel affine transformation, noise addition, intensity transformation, and blurring processing on the image sample to add disturbance to the image sample.

[0174] The first model training unit 1005 is configured to train the first key point embedding sub-model using the first training data, wherein the first key point embedding sub-model performs feature extraction of one or more scales on each key point in the input first image sample and each key point in the second image sample to obtain feature representations of one or more scales corresponding to each key point in the first image sample and feature representations of one or more scales corresponding to each key point in the second image sample; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the first positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the first negative sample pair.

[0175] The registration processing unit 1006 is configured to register the initial first modality image and the second modality image using the first key point embedding sub-model trained by the first model training unit 1005 .

[0176] The second sample acquisition unit 1001 obtains second training data using the first modality image and the second modality image registered by the registration processing unit 1006 .

[0177] As one of the implementable ways, the registration processing unit 1006 can also be configured to register the first modality image and the second modality image using the second key point embedding sub-model currently trained by the second model training unit 1002. The second sample acquisition unit 1001 updates the second training data using the first modality image and the second modality image registered by the registration processing unit 1006; the second model training unit 1002 continues to train the second key point embedding sub-model using the second training data until a preset iteration stop condition is met.

[0178] Correspondingly, the model building unit 1003 obtains a key point matching model using the second key point embedding sub-model and the matching sub-model obtained by the second model training unit 1002 after the preset iteration stopping condition is met.

[0179] As one of the achievable methods, the registration processing unit 1006 can be specifically configured to determine multiple matching key point pairs in the first modality image and the second modality image using feature representations of more than one scale corresponding to each key point in the first modality image and the second modality image; determine the transformation matrix between the first modality image and the second modality image using the multiple matching key point pairs; and transform the second modality image using the transformation matrix to obtain the second modality image registered with the first modality image.

[0180] According to yet another embodiment, a cross-modal key point matching device is provided. Fig.11 FIG. 1 is a schematic block diagram of a cross-modal key point matching device according to an embodiment. Fig.11 As shown, the device 1100 includes: an image acquisition unit 1101 and a key point matching unit 1102. The main functions of each component unit are as follows:

[0181] The image acquisition unit 1101 is configured to acquire a first modality image and a second modality image to be matched, where the first modality image and the second modality image are acquired for the same type of object;

[0182] The key point matching unit 1102 is configured to input the first modality image and the second modality image to be matched into the Fig.10 The device shown pre-establishes a key point matching model, which includes a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be matched; the matching sub-model uses the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be matched, to match the key points in the first modality image and the second modality image to be matched.

[0183] According to yet another embodiment, a cross-modality image registration apparatus is provided. Fig.12 FIG. 1 is a schematic block diagram of a cross-modal image registration apparatus according to an embodiment. Fig.12 As shown, the device 1200 includes: an image acquisition unit 2101, a key point matching unit 1202 and an image registration unit 1203. The main functions of each component unit are as follows:

[0184] The image acquisition unit 1201 is configured to acquire a first modality image and a second modality image to be registered, where the first modality image and the second modality image are acquired for the same type of object.

[0185] The key point matching unit 1202 is configured to input the first modality image and the second modality image to be registered into a key point matching model pre-established by the device of claim 10, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively; the matching sub-model matches the key points in the first modality image and the second modality image to be registered using the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered respectively.

[0186] The image registration unit 1203 is configured to register the first modality image to be registered and the second modality image using the multiple matching key point pairs obtained by matching.

[0187] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0189] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0190] And an electronic device, comprising:

[0191] one or more processors; and

[0192] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0193] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0194] in, Fig.13 The architecture of the electronic device is shown as an example, which may include a processor 1310, a video display adapter 1311, a disk drive 1312, an input / output interface 1313, a network interface 1314, and a memory 1320. The processor 1310, the video display adapter 1311, the disk drive 1312, the input / output interface 1313, the network interface 1314, and the memory 1320 may be communicatively connected via a communication bus 1330.

[0195] Among them, the processor 1310 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0196] The memory 1320 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1320 can store an operating system 1321 for controlling the operation of the electronic device 1300, and a basic input and output system (BIOS) 1322 for controlling the low-level operation of the electronic device 1300. In addition, a web browser 1323, a data storage management system 1324, and a model training device / key point matching device / image registration device 1325, etc. can also be stored. The above-mentioned model training device / key point matching device / image registration device 1325 can be an application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 1320 and is called and executed by the processor 1310.

[0197] The input / output interface 1313 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0198] The network interface 1314 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0199] The bus 1330 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1310, the video display adapter 1311, the disk drive 1312, the input / output interface 1313, the network interface 1314, and the memory 1320).

[0200] It should be noted that, although the above device only shows a processor 1310, a video display adapter 1311, a disk drive 1312, an input / output interface 1313, a network interface 1314, a memory 1320, a bus 1330, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0201] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0202] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for establishing a key point matching model, characterized in that: The method comprises: Acquire second training data including a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to positions of the same object collected from the first modality image and the second modality image, respectively, and the second negative sample pairs include key points corresponding to positions of different objects collected from the first modality image and the second modality image, respectively, and the first modality image and the second modality image are collected for the same object; The second key point embedding submodel is trained using the second training data, wherein the second key point embedding submodel performs feature extraction of one or more scales on each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair; A key point matching model is obtained using the trained second key point embedding submodel and the matching submodel, wherein the matching submodel is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding submodel.

2. The method according to claim 1, characterized in that: Before obtaining the second training data including a plurality of positive sample pairs and negative sample pairs, the method further includes: Acquire first training data including a plurality of first positive sample pairs and first negative sample pairs, wherein the first positive sample pairs include key points corresponding to the same object position collected from the first image sample and the second image sample, respectively, and the first negative sample pairs include key points corresponding to different object positions collected from the first image sample and the second image sample, respectively; wherein the first image sample and the second image sample are images obtained by adding different disturbances to the same image sample, or the first image sample and the second image sample are an image sample and an image obtained by adding disturbances to the image sample, respectively; The first training data is used to train a first key point embedding submodel, wherein the first key point embedding submodel performs feature extraction of one or more scales on each key point in the input first image sample and each key point in the second image sample to obtain feature representations of one or more scales corresponding to each key point in the first image sample and feature representations of one or more scales corresponding to each key point in the second image sample; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the first positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the first negative sample pair; The initial first modality image and the second modality image are registered using the trained first key point embedding sub-model, and the second training data is obtained using the registered first modality image and the second modality image.

3. The method according to claim 1, characterized in that After the second key point embedding submodel is trained using the second training data, the method further includes: registering the first modality image and the second modality image using the second key point embedding submodel obtained by the current training, and updating the second training data using the registered first modality image and the second modality image; and continuing to execute the step of training the second key point embedding submodel using the second training data until a preset iteration stop condition is met; The method of obtaining a key point matching model by using the second key point embedding sub-model and the matching sub-model obtained through training includes: obtaining a key point matching model by using the second key point embedding sub-model and the matching sub-model obtained after satisfying a preset iteration stop condition.

4. The method according to claim 2 or 3, characterized in that: The registration includes: Determine a plurality of matching key point pairs in the first modality image and the second modality image using feature representations of one or more scales corresponding to each key point in the first modality image and the second modality image; Determining a transformation matrix between the first modality image and the second modality image using the plurality of matching key point pairs; The second modality image is transformed using the transformation matrix to obtain the second modality image registered with the first modality image.

5. The method according to claim 2, characterized in that: Adding disturbance to the image sample includes: performing at least one of super-pixel affine transformation, adding noise, intensity transformation and blurring processing on the image sample.

6. A cross-modal key point matching method, characterized in that: The method comprises: Acquire a first modality image and a second modality image to be matched, wherein the first modality image and the second modality image are acquired for the same type of object; Inputting the first modality image and the second modality image to be matched into a key point matching model pre-established by the method according to any one of claims 1 to 5, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; The second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be matched respectively; The matching sub-model uses feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be matched, to match the key points in the first modality image and the second modality image to be matched.

7. A cross-modal image registration method, characterized in that: The method comprises: Acquire a first modality image and a second modality image to be registered, wherein the first modality image and the second modality image are acquired for the same type of object; Inputting the first modality image and the second modality image to be registered into a key point matching model pre-established by the method of any one of claims 1 to 5, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; The second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively; The matching sub-model uses feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered, to match key points in the first modality image and the second modality image to be registered; The first modality image and the second modality image to be registered are registered using the multiple matching key point pairs obtained by the matching.

8. The method according to claim 7, characterized in that The method is applied to a medical scenario, and the first modality image and the second modality image to be registered include: a first modality medical image and a second modality medical image acquired for a body part of an animal of the same type.

9. A cross-modal image registration method, characterized in that: The method comprises: Displaying the first modality image and the second modality image to be registered on a display screen, a virtual reality VR device, or an augmented display AR device, where the first modality image and the second modality image are acquired for the same type of object; Inputting the first modality image and the second modality image to be registered into a key point matching model pre-established by the method of any one of claims 1 to 5, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; The second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively; The matching sub-model uses feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered, to match key points in the first modality image and the second modality image to be registered; Using the multiple matching key point pairs obtained by the matching, registering the first modality image and the second modality image to be registered; The display screen, VR device or AR device is driven to render and display the registered first modality image and second modality image.

10. A device for establishing a key point matching model, characterized in that: The device comprises: a second sample acquisition unit, configured to acquire second training data comprising a plurality of second positive sample pairs and second negative sample pairs, wherein the second positive sample pairs include key points corresponding to positions of the same object collected from the first modality image and the second modality image, respectively; and the second negative sample pairs include key points corresponding to positions of different objects collected from the first modality image and the second modality image, respectively; and the first modality image and the second modality image are collected for the same object; The second model training unit is configured to train a second key point embedding submodel using the second training data, wherein the second key point embedding submodel performs feature extraction of one or more scales on each key point in the input first modality image and each key point in the second modality image, respectively, to obtain feature representations of one or more scales corresponding to each key point in the first modality image and feature representations of one or more scales corresponding to each key point in the second modality image; the training objectives include: minimizing the distance between the feature representations of one or more scales corresponding to two key points in the second positive sample pair, and maximizing the distance between the feature representations of one or more scales corresponding to two key points in the second negative sample pair; The model building unit is configured to obtain a key point matching model using the second key point embedding sub-model and the matching sub-model trained by the second model training unit, wherein the matching sub-model is used to match the key points in the images of the two modalities using feature representations of one or more scales of each key point in the images of the two modalities extracted by the second key point embedding sub-model.

11. A cross-modal key point matching device, characterized in that: The device comprises: An image acquisition unit is configured to acquire a first modality image and a second modality image to be matched, wherein the first modality image and the second modality image are acquired for the same type of object; A key point matching unit is configured to input the first modality image and the second modality image to be matched into a key point matching model pre-established by the device according to claim 10, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be matched respectively; the matching sub-model matches the key points in the first modality image and the second modality image to be matched using the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be matched respectively.

12. A cross-modal image registration device, characterized in that: The device comprises: An image acquisition unit is configured to acquire a first modality image and a second modality image to be registered, wherein the first modality image and the second modality image are acquired for the same type of object; A key point matching unit, configured to input the first modality image and the second modality image to be registered into a key point matching model pre-established by the device according to claim 10, wherein the key point matching model includes a second key point embedding sub-model and a matching sub-model; the second key point embedding sub-model extracts feature representations of one or more scales corresponding to each key point from the first modality image and the second modality image to be registered respectively; the matching sub-model uses the feature representations of one or more scales corresponding to each key point extracted from the first modality image and the second modality image to be registered respectively to match the key points in the first modality image and the second modality image to be registered; The image registration unit is configured to register the first modality image and the second modality image to be registered using the multiple matching key point pairs obtained by the matching.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 9.

Images