Image registration method and device, electronic equipment and storage medium

Abstract

The application discloses an image registration method and device, electronic equipment and storage medium. The method comprises the following steps: obtaining a first image and a second image to be registered; obtaining a pre-trained feature extraction model, and determining a first feature image corresponding to the first image and a second feature image corresponding to the second image based on the feature extraction model; obtaining a pre-trained registration model, and determining a segmentation map of a first region of interest and a position transformation matrix of a second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model. Through the technical scheme disclosed by the application, the problem that the existing technology cannot follow up the remaining lesions after surgery is solved, and the two images with large differences are directly registered, so that accurate follow-up of unexcised lesions is realized under the condition of preoperative and postoperative lesion excision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image registration method and device, electronic equipment and a storage medium. BACKGROUND

[0002] At present, all lesions that have not been operated on are matched for registration, so as to realize follow-up of unexcised lesions. Because the change of anatomical structure caused by surgery and the excision of some related areas will reduce the registration accuracy. For example, patients have acute lung infection diseases such as new coronary pneumonia or related complications, and when the patient is followed up after surgery, the existing method cannot achieve the required accuracy. SUMMARY

[0003] The present application provides an image registration method, device, electronic equipment and storage medium to solve the problem that the existing technology cannot follow up the lesions that still exist after surgery, and directly registers two images with large differences, thereby realizing accurate follow-up of unexcised lesions in the case of preoperative and postoperative lesion excision.

[0004] In a first aspect, an image registration method is provided, which comprises:

[0005] Obtaining a first image and a second image to be registered;

[0006] Obtaining a pre-trained feature extraction model, and determining a first feature image corresponding to the first image and a second feature image corresponding to the second image based on the feature extraction model;

[0007] Obtaining a pre-trained registration model, and determining a segmentation map of a first region of interest and a position transformation matrix of a second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model.

[0008] Optionally, the feature extraction model comprises a shallow feature extraction module and a deep feature extraction module, the first feature image comprises a first shallow feature image and a first deep feature image, and the second feature image comprises a second shallow feature image and a second deep feature image;

[0009] Correspondingly, the determination of the first feature image corresponding to the first image and the second feature image corresponding to the second image based on the feature extraction model comprises:

[0010] The first image is input into the shallow feature extraction module and the deep feature extraction module respectively to obtain the first shallow feature image and the first deep feature image of the first image;

[0011] inputting the second image into the shallow feature extraction module and the deep feature extraction module respectively to obtain a second shallow feature image and a second deep feature image of the second image.

[0012] Optionally, the determining of the segmentation map of the first region of interest and the position transformation matrix of the second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model comprises:

[0013] inputting the first shallow feature image and the second shallow feature image into the registration model to obtain a segmentation map of a first shallow region of interest and a position transformation matrix of a shallow second region of interest output by the registration model;

[0014] inputting the first deep feature image and the second deep feature image into the registration model to obtain a segmentation map of a deep first region of interest and a position transformation matrix of a deep second region of interest output by the registration model;

[0015] determining the segmentation map of the first region of interest between the first image and the second image based on the segmentation map of the first shallow region of interest and the segmentation map of the deep first region of interest, and determining the position transformation matrix of the second region of interest between the first image and the second image based on the position transformation matrix of the shallow second region of interest and the position transformation matrix of the deep second region of interest.

[0016] Optionally, the first image is a preoperative image of a preset part, the second image is a postoperative image corresponding to the preset part, the segmentation map of the first region of interest between the preoperative image and the postoperative image is a lesion region removed by surgery, and the position transformation matrix of the second region of interest is a position transformation relationship of other lesion regions of the preset part before and after surgery.

[0017] Optionally, the training method of the registration model comprises:

[0018] obtaining an initial position transformation matrix of a second region of interest and an initial segmentation map of a first region of interest of the registration model in training;

[0019] obtaining a first sample image and a second sample image with completed sample labeling;

[0020] generating a first sample transformation image based on the first sample image, the initial position transformation matrix of the second region of interest and the initial segmentation map of the first region of interest, and generating a first loss function based on the first sample transformation image and the second sample image;

[0021] adjust the model parameters of the registration model in the current iteration based on the first loss function, and continue to perform iterative training until an iterative stop condition is met to obtain a trained registration model.

[0022] Optionally, the training method of the registration model further comprises:

[0023] obtaining a first sample feature image corresponding to the first sample image and a second sample feature image corresponding to the second sample image based on a pre-trained feature extraction model;

[0024] generating a first sample feature transformation image based on the first sample feature image, a position transformation matrix of the initial second region of interest, and a segmentation map of the initial first region of interest, and generating a second loss function based on the first sample feature transformation image and the second sample feature image;

[0025] generating a model loss function based on the first loss function and the second loss function, and adjusting the model parameters of the registration model in the current iteration based on the model loss function, and continuing to perform iterative training until an iterative stop condition is met to obtain a trained registration model.

[0026] Optionally, the first sample image is a preoperative image of a preset part, and the second sample image is a postoperative image corresponding to the preset part.

[0027] The sample annotations include first sample annotations of resected lesions in the preoperative image and the postoperative image, and second sample annotations of non-resected lesions in the preset part.

[0028] The model loss function includes a local loss function generated based on the first sample annotations and a global loss function generated based on the second sample annotations.

[0029] In a second aspect, an embodiment of the present application further provides an image registration device, which comprises:

[0030] an image acquisition module, configured to acquire a first image and a second image to be registered;

[0031] a feature image acquisition module, configured to acquire a pre-trained feature extraction model, and determine a first feature image corresponding to the first image and a second feature image corresponding to the second image based on the feature extraction model;

[0032] An image registration module is configured to acquire a pre-trained registration model, and determine a segmentation map of a first region of interest and a position transformation matrix of a second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model.

[0033] In a third aspect, an electronic device is provided, comprising:

[0034] at least one processor; and

[0035] a memory in communication with the at least one processor; wherein

[0036] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the image registration method according to any one of the embodiments of the present application.

[0037] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions for enabling a processor to implement the image registration method according to any one of the embodiments of the present application.

[0038] The technical scheme provided by the embodiments of the present application can accurately follow up by determining the change of the existing lesion in the preset time interval on the basis of the lesion region removed by surgery and other lesions existing in the preset region. In the above scheme, the similarity is limited by the deep and shallow features of the image, the features related to registration are more comprehensively extracted, and the accuracy of the registration result based on the features is improved. The problem that the registration result is inaccurate and the lesion matching and positioning are ineffective due to the large difference between the preoperative and postoperative anatomic structures is solved. The two images with large differences are directly registered, and the unremoved lesion is accurately followed up in the case of preoperative and postoperative lesion removal.

[0039] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 is a flow chart of an image registration method according to an embodiment of the present application;

[0042] Figure 2 is a structural schematic diagram of an image registration device according to an embodiment of the present application;

[0043] Figure 3 is a structural schematic diagram of an electronic device implementing an image registration method according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0045] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0046] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0047] It can be understood that, before using the technical solutions disclosed in the embodiments of the present application, the type, use range, use scenario, etc. of the personal information involved in the present application should be informed to the user and the authorization of the user should be obtained according to relevant laws and regulations.

[0048] For example, when responding to the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be performed will require the acquisition and use of the personal information of the user. Thus, the user can voluntarily choose whether to provide the personal information to the electronic device, application program, server or storage medium, etc. software or hardware performing the operation of the technical solutions of the present application according to the prompt information.

[0049] As an optional but non-limiting implementation, in response to receiving the active request of the user, the manner of sending the prompt information to the user may be, for example, a pop-up window manner, in which the prompt information may be presented in the form of text. In addition, the pop-up window may also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0050] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the present disclosure, and other manners that meet the relevant laws and regulations can also be applied to the implementation of the present disclosure.

[0051] It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and the relevant provisions.

[0052] Embodiment one

[0053] Figure 1 A flowchart of an image registration method is provided for the first embodiment of the present application. The present embodiment can be applied to the registration of two images obtained by scanning the same object at different times. The method can be performed by an image registration device, which can be implemented in the form of hardware and / or software. The image registration device can be configured in a smart terminal and a cloud server. As shown in the figure, the method comprises: Figure 1

[0054] S110, obtaining a first image and a second image to be registered.

[0055] In the embodiment of the present application, the first image and the second image can be understood as two images obtained by photographing a preset region of the same target object at a preset time interval. In order to determine the change of the preset region of the target object within the preset time interval, the two images need to be registered.

[0056] Specifically, the manner of obtaining the first image and the second image is different according to different target objects. Optionally, if the target object is a patient with a disease, the manner of obtaining the image can be to use a medical imaging device to scan the patient with a disease at a first time and a second time which is separated from the first time by a preset time interval, thereby obtaining the first image and the second image. Optionally, if the target object is a static natural landscape, such as a stone or a tree, the manner of obtaining the image can be to use a camera or other photographing device to scan the target object such as a stone or a tree at a first time and a second time which is separated from the first time by a preset time interval, thereby obtaining the first image and the second image.

[0057] ​In the embodiment of the present application, in order to realize accurate registration of two images, image features of the two images can be acquired respectively, and registration results can be obtained based on image registration and feature registration, so as to improve the accuracy of the registration results.

[0058] In S120, a feature extraction model trained in advance is acquired, and a first feature image corresponding to the first image and a second feature image corresponding to the second image are determined based on the feature extraction model.

[0059] In the embodiment of the present application, the feature extraction model can be a feature extraction model obtained based on training of a Unet network structure, and of course, can be based on other network structures. The network structure and training method of the feature extraction model are not limited in the embodiment.

[0060] Specifically, the image is subjected to feature extraction processing based on the trained feature extraction model, and a first feature image corresponding to the first image and a second feature image corresponding to the second image are obtained respectively.

[0061] In the embodiment, the feature extraction model includes a shallow feature extraction module and a deep feature extraction module, the first feature image includes a first shallow feature image and a first deep feature image, and the second feature image includes a second shallow feature image and a second deep feature image.

[0062] Correspondingly, in the embodiment, the first feature image corresponding to the first image and the second feature image corresponding to the second image are determined based on the feature extraction model, including: inputting the first image into the shallow feature extraction module and the deep feature extraction module respectively to obtain the first shallow feature image and the first deep feature image of the first image; and inputting the second image into the shallow feature extraction module and the deep feature extraction module respectively to obtain the second shallow feature image and the second deep feature image of the second image.

[0063] Specifically, the first image is input into the shallow feature extraction module of the feature extraction model to obtain the first shallow feature image corresponding to the first image, and then the first shallow feature image is input into the deep feature extraction module of the feature extraction model to obtain the first deep feature image corresponding to the first image.

[0064] Similarly, the second image is input into the shallow feature extraction module of the feature extraction model to obtain the second shallow feature image corresponding to the second image, and then the second shallow feature image is input into the deep feature extraction module of the feature extraction model to obtain the second deep feature image corresponding to the second image.

[0065] Optionally, the method for obtaining the image features in the embodiment can be based on a pre-trained feature extraction model, and of course, the image can be processed based on a preset image algorithm calculation formula to obtain the image features of the image, and the feature extraction manner is not limited.

[0066] In S130, a pre-trained registration model is obtained, and based on the first feature image, the second feature image and the registration model, a segmentation map of a first region of interest and a position transformation matrix of a second region of interest between the first image and the second image are determined.

[0067] In the embodiment of the application, the registration model can be a registration model obtained based on training of network structures such as a dilated convolution and a deformable convolution, and of course, can be based on other network structures, and the network structure and the training manner of the registration model are not limited in the embodiment.

[0068] In order to obtain accurate registration results, the technical solution of the embodiment needs to pre-train the registration model before registration processing to obtain a trained registration model to realize subsequent accurate registration.

[0069] Optionally, the training method of the registration model in the embodiment includes: obtaining an initial position transformation matrix of a second region of interest and an initial segmentation map of a first region of interest of the registration model in training; obtaining a first sample image and a second sample image with completed sample labeling; generating a first sample transformation image based on the first sample image, the initial position transformation matrix of the second region of interest and the initial segmentation map of the first region of interest, and generating a first loss function based on the first sample transformation image and the second sample image; adjusting the model parameters of the registration model in the current iteration based on the first loss function, and continuing to perform iterative training until the trained registration model is obtained by meeting the iteration stop condition.

[0070] Optionally, in order to obtain a registration model with better performance, on the basis of the above training method, the embodiment further provides another training method, specifically, on the basis of generating the first loss function, the first sample feature image corresponding to the first sample image and the second sample feature image corresponding to the second sample image are obtained based on a pre-trained feature extraction model; the first sample feature transformation image is generated based on the first sample feature image, the initial position transformation matrix of the second region of interest and the initial segmentation map of the first region of interest, and the second loss function is generated based on the first sample feature transformation image and the second sample feature image; the model loss function is generated based on the first loss function and the second loss function, and the model parameters of the registration model are adjusted in the current iteration based on the model loss function, and the iterative training is continued until the trained registration model is obtained by meeting the iteration stop condition.

[0071] The method of training the registration model based on the sample image and the sample feature image can improve the learning ability of the registration model based on the feature image for registration, so that the subsequent registration model has better registration effect.

[0072] Optionally, in the process of training the registration model, the first sample image can be a preoperative image of a preset part, and the second sample image can be a postoperative image corresponding to the preset part; accordingly, the sample label includes a first sample label of a resected lesion in the preoperative image and a second sample label of a non-resected lesion in the preset part; and the model loss function includes a local loss function generated based on the first sample label and a global loss function generated based on the second sample label.

[0073] Specifically, the first loss function generated based on the first image and the second image includes a local loss function generated based on the first sample label and a global loss function generated based on the second sample label. Specifically, the global loss function is a pixel value similarity loss and a structural similarity loss of a part other than the part labeled as a postoperative resection part. The local loss function is obtained from the labeled matching lesion. In addition to the global loss function, the similarity of the region where the lesion exists and the corresponding condition of the matching lesion need to be additionally focused on, so that the registration model trained has higher accuracy when performing registration processing.

[0074] Specifically, the loss function generated based on the first feature image and the second feature image also includes a local loss function generated based on the first sample label and a global loss function generated based on the second sample label. The difference is that the loss function obtained based on the feature image uses the average cosine similarity loss as the loss function.

[0075] On the basis of the above-mentioned embodiments, the technical scheme of the present embodiment can also use the intersection over union loss or the cross-entropy loss to determine the first sample label loss function of the part labeled as the postoperative resection part, and use the position regression loss function to calculate the second sample label loss function of the corresponding lesion point, so as to improve the learning ability of the registration model and obtain a registration model with better effect.

[0076] It should be further noted that in the training process of the present embodiment, the image changes before and after the operation can be large, and a large position change can occur. Therefore, a large position change is allowed, but the position change of the adjacent pixels must be similar, so as to limit the change space of each pixel point. The gradient of the transformation coordinates of the xyz three directions of the position transformation matrix is solved first, and then the gradient size is limited to be not too large, so as to limit the position change similarity.

[0077] To improve the accuracy of image registration, the technical solution of the embodiment simultaneously registers the first shallow feature image and the second shallow feature image, and registers the first deep feature image and the second deep feature image, and determines the registration result of the first image and the second image based on the registration results of the above. In the embodiment, the registration result includes the segmentation map of the first region of interest and the position transformation matrix of the second region of interest. Specifically, the region of interest is used to represent the region that exists in the first image but does not exist in the second image, the second region of interest is used to represent the region of interest that exists in both the first image and the second image, and the position transformation matrix of the second region of interest is used to represent the change of the second region of interest within a preset time interval.

[0078] Optionally, the method for determining the segmentation map of the first region of interest and the position transformation matrix of the second region of interest between the first image and the second image based on the first feature image, the second feature image, and the registration model in the embodiment can include: inputting the first shallow feature image and the second shallow feature image into the registration model to obtain the segmentation map of the shallow first region of interest and the position transformation matrix of the shallow second region of interest output by the registration model; inputting the first deep feature image and the second deep feature image into the registration model to obtain the segmentation map of the deep first region of interest and the position transformation matrix of the deep second region of interest output by the registration model; determining the segmentation map of the first region of interest between the first image and the second image based on the segmentation map of the shallow first region of interest and the segmentation map of the deep first region of interest, and determining the position transformation matrix of the second region of interest between the first image and the second image based on the position transformation matrix of the shallow second region of interest and the position transformation matrix of the deep second region of interest.

[0079] Specifically, the obtained segmentation map of the shallow first region of interest can be determined as the segmentation map of the first region of interest between the first image and the second image, and the position transformation matrix of the shallow second region of interest can be determined as the position transformation matrix of the second region of interest between the first image and the second image.

[0080] Optionally, the obtained segmentation map of the deep first region of interest can also be determined as the segmentation map of the first region of interest between the first image and the second image, and the position transformation matrix of the deep second region of interest can also be determined as the position transformation matrix of the second region of interest between the first image and the second image.

[0081] Optionally, the technical scheme of the embodiment can also perform weighted average processing on the obtained segmentation map of the shallow first region of interest and the segmentation map of the deep first region of interest, and take the segmentation map of the first region of interest obtained after the processing as the segmentation map of the first region of interest between the first image and the second image; perform weighted average processing on the position transformation matrix of the shallow second region of interest and the position transformation matrix of the deep second region of interest, and determine the position transformation matrix of the region of interest obtained after the processing as the position transformation matrix of the second region of interest between the first image and the second image.

[0082] On the basis of the above-mentioned embodiment, in the embodiment, the first image is a preoperative image of a preset part, the second image is a postoperative image corresponding to the preset part, the segmentation map of the first region of interest between the preoperative image and the postoperative image is a segmentation map of a lesion region removed by surgery, and the position transformation matrix of the second region of interest is a position transformation relationship of other lesion regions of the preset part before and after the surgery.

[0083] The technical scheme provided in the embodiment can accurately follow up by first determining a lesion region removed by surgery and other lesions still existing in the preset region, and then determining the change of the existing lesion within a preset time interval, thereby achieving accurate follow-up. In the above scheme, the similarity is limited by the deep features and the shallow features of the image together, the features related to registration are more comprehensively extracted, and the accuracy of the registration result based on the features is improved, thereby solving the problem that the anatomic structure changes before and after the surgery cause a large difference in the image data, thereby causing inaccurate registration results and invalid lesion matching and positioning, and achieving direct registration of two images with a large difference, thereby achieving accurate follow-up of the lesion not removed by the surgery under the condition that the lesion is removed before and after the surgery.

[0084] Embodiment two

[0085] Figure 2 A structural schematic diagram of an image registration device provided in the embodiment three of the application is shown in FIG. 3. Figure 2 As shown in the figure, the device includes an image acquisition module 210, a feature image acquisition module 220, and an image registration module 230; wherein,

[0086] The image acquisition module 210 is configured to acquire a second image to be registered and the second image.

[0087] The feature image acquisition module 220 is configured to acquire a pre-trained feature extraction model, and determine a second feature image corresponding to the second image and the second feature image corresponding to the second image based on the feature extraction model.

[0088] The image registration module 230 is configured to acquire a pre-trained registration model, and determine a segmentation map of a first region of interest between the second image and the second image and a position transformation matrix of a second region of interest between the second image and the second image based on the second feature image, the second feature image and the registration model.

[0089] On the basis of the above embodiments, optionally, the feature extraction model comprises a shallow feature extraction module and a deep feature extraction module, the second feature image comprises a second shallow feature image and a second deep feature image, and the second feature image comprises a second shallow feature image and a second deep feature image.

[0090] Correspondingly, the feature image acquisition module 220 comprises:

[0091] The second feature image obtaining unit is configured to input the second image into the shallow feature extraction module and the deep feature extraction module respectively to obtain a second shallow feature image and a second deep feature image of the second image.

[0092] The second feature image obtaining unit is configured to input the second image into the shallow feature extraction module and the deep feature extraction module respectively to obtain a second shallow feature image and a second deep feature image of the second image.

[0093] On the basis of the above embodiments, optionally, the image registration module 230 comprises:

[0094] The second registration result obtaining unit is configured to input the second shallow feature image and the second shallow feature image into the registration model to obtain a segmentation map of a first shallow region of interest and a position transformation matrix of a shallow second region of interest output by the registration model.

[0095] The second registration result obtaining unit is configured to input the second deep feature image and the second deep feature image into the registration model to obtain a segmentation map of a deep first region of interest and a position transformation matrix of a deep second region of interest output by the registration model.

[0096] The registration result obtaining unit is configured to determine a segmentation map of a first region of interest between the second image and the second image based on the segmentation map of the first shallow region of interest and the segmentation map of the deep first region of interest, and determine a position transformation matrix of a second region of interest between the second image and the second image based on the position transformation matrix of the shallow second region of interest and the position transformation matrix of the deep second region of interest.

[0097] On the basis of each of the above embodiments, optionally, the second image is a preoperative image of a preset part, the second image is a postoperative image corresponding to the preset part, and a segmentation map of a first region of interest between the preoperative image and the postoperative image is a lesion region removed by surgery; and the position transformation matrix of the second region of interest is a position transformation relationship of other lesion regions of the preset part before and after surgery.

[0098] On the basis of each of the above embodiments, optionally, the device further comprises a second model training module; the second model training module is configured to obtain an initial position transformation matrix of a second region of interest of a registration model in training and a segmentation map of an initial first region of interest; obtain a second sample image with sample labeling completed and a second sample image; generate a second sample transformed image based on the second sample image, the initial position transformation matrix of the second region of interest, and the segmentation map of the initial first region of interest, and generate a second loss function based on the second sample transformed image and the second sample image; adjust model parameters of the registration model in the current iteration based on the second loss function, and continue to perform iterative training until an iterative stop condition is met to obtain a trained registration model.

[0099] On the basis of each of the above embodiments, optionally, the device further comprises a second model training module; the second model training module is configured to obtain a second sample feature image corresponding to the second sample image and a second sample feature image corresponding to the second sample image based on a pre-trained feature extraction model; generate a second sample feature transformed image based on the second sample feature image, the initial position transformation matrix of the second region of interest, and the segmentation map of the initial first region of interest, and generate a second loss function based on the second sample feature transformed image and the second sample feature image; generate a model loss function based on the second loss function and the second loss function, and adjust model parameters of the registration model in the current iteration based on the model loss function, and continue to perform iterative training until an iterative stop condition is met to obtain a trained registration model.

[0100] On the basis of each of the above embodiments, optionally, the second sample image is a preoperative image of a preset part, and the second sample image is a postoperative image corresponding to the preset part.

[0101] The sample labeling includes second sample labeling of a resected lesion in the preoperative image and the postoperative image and second sample labeling of a non-resected lesion in the preset part.

[0102] The model loss function includes a local loss function generated based on the second sample labeling and a global loss function generated based on the second sample labeling.

[0103] The image registration apparatus provided by the embodiments of the present application can perform the image registration method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method.

[0104] Embodiment Three

[0105] Figure 3 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0106] As shown in Figure 3 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0107] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, a loudspeaker, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0108] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, and the like. The processor 11 performs various methods and processes described above, such as the image registration method.

[0109] In some embodiments, the image registration method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the image registration method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the image registration method by any other suitable means, such as by means of firmware.

[0110] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0111] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0112] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0115] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0116] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0117] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image registration method characterized by, The method comprises the following steps: obtaining a first image and a second image to be registered; obtaining a pre-trained feature extraction model, and determining a first feature image corresponding to the first image and a second feature image corresponding to the second image based on the feature extraction model; obtaining a pre-trained registration model, and determining a segmentation map of a first region of interest and a position transformation matrix of a second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model; the position transformation matrix of the second region of interest is the position transformation relationship of other lesion regions of a preset part before and after surgery; the feature extraction model comprises a shallow feature extraction module and a deep feature extraction module, the first feature image comprises a first shallow feature image and a first deep feature image, and the second feature image comprises a second shallow feature image and a second deep feature image; determining the segmentation map of the first region of interest and the position transformation matrix of the second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model comprises: inputting the first shallow feature image and the second shallow feature image into the registration model to obtain a shallow first region of interest segmentation map and a shallow second region of interest position transformation matrix output by the registration model; inputting the first deep feature image and the second deep feature image into the registration model to obtain a deep first region of interest segmentation map and a deep second region of interest position transformation matrix output by the registration model; determining the segmentation map of the first region of interest between the first image and the second image based on the shallow first region of interest segmentation map and the deep first region of interest segmentation map, and determining the position transformation matrix of the second region of interest between the first image and the second image based on the shallow second region of interest position transformation matrix and the deep second region of interest position transformation matrix; the first image is a preoperative image of a preset part, the second image is a postoperative image corresponding to the preset part, and the segmentation map of the first region of interest between the preoperative image and the postoperative image is a lesion region removed by surgery.

2. The method of claim 1, wherein obtaining a pre-trained feature extraction model, and determining a first feature image corresponding to the first image and a second feature image corresponding to the second image based on the feature extraction model comprises: inputting the first image into the shallow feature extraction module and the deep feature extraction module respectively to obtain a first shallow feature image and a first deep feature image of the first image; inputting the second image into the shallow feature extraction module and the deep feature extraction module respectively to obtain a second shallow feature image and a second deep feature image of the second image.

3. The method of claim 1, wherein, The training method of the registration model comprises: obtain a position transformation matrix of an initial second region of interest and a segmentation map of an initial first region of interest of the registration model in training; obtain a first sample image and a second sample image with sample labeling completed; generate a first sample transformed image based on the first sample image, the position transformation matrix of the initial second region of interest and the segmentation map of the initial first region of interest, and generate a first loss function based on the first sample transformed image and the second sample image; adjust the model parameters of the registration model in the current iteration based on the first loss function, and continue to perform iterative training until the iterative stop condition is met to obtain the trained registration model.

4. The method of claim 3, wherein, The training method of the registration model further comprises: obtain a first sample feature image corresponding to the first sample image and a second sample feature image corresponding to the second sample image based on the pre-trained feature extraction model; generate a first sample feature transformed image based on the first sample feature image, the position transformation matrix of the initial second region of interest and the segmentation map of the initial first region of interest, and generate a second loss function based on the first sample feature transformed image and the second sample feature image; generate a model loss function based on the first loss function and the second loss function, and adjust the model parameters of the registration model in the current iteration based on the model loss function, and continue to perform iterative training until the iterative stop condition is met to obtain the trained registration model.

5. The method according to any of claims 3-4, characterized by, The first sample image is a preoperative image of a preset part, and the second sample image is a postoperative image corresponding to the preset part; The sample labeling includes first sample labeling of resected lesions in the preoperative image and the postoperative image, and second sample labeling of non-resected lesions in the preset part; The model loss function includes a local loss function generated based on the first sample labeling and a global loss function generated based on the second sample labeling.

6. An image registration apparatus characterized by comprising: It comprises: an image acquisition module configured to acquire a first image and a second image to be registered; a feature image acquisition module configured to acquire a pre-trained feature extraction model, and determine a first feature image corresponding to the first image and a second feature image corresponding to the second image based on the feature extraction model; an image registration module configured to acquire a pre-trained registration model, determine a segmentation map of a first region of interest and a position transformation matrix of a second region of interest between the first image and the second image based on the first feature image, the second feature image and the registration model; The position transformation matrix of the second region of interest is the position transformation relationship of other lesion regions of a preset part before and after operation; The feature extraction model comprises a shallow feature extraction module and a deep feature extraction module, the first feature image comprises a first shallow feature image and a first deep feature image, and the second feature image comprises a second shallow feature image and a second deep feature image; The image registration module comprises: The first registration result obtaining unit is configured to input the first shallow feature image and the second shallow feature image into the registration model to obtain a segmentation map of a shallow first region of interest and a position transformation matrix of a shallow second region of interest output by the registration model; The second registration result obtaining unit is configured to input the first deep feature image and the second deep feature image into the registration model to obtain a segmentation map of a deep first region of interest and a position transformation matrix of a deep second region of interest output by the registration model; The registration result obtaining unit is configured to determine a segmentation map of a first region of interest between the first image and the second image based on the segmentation map of the shallow first region of interest and the segmentation map of the deep first region of interest, and determine a position transformation matrix of a second region of interest between the first image and the second image based on the position transformation matrix of the shallow second region of interest and the position transformation matrix of the deep second region of interest. The first image is a preoperative image of a preset part, the second image is a postoperative image corresponding to the preset part, and the segmentation map of the first region of interest between the preoperative image and the postoperative image is a lesion region removed by surgery.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image registration method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to execute the image registration method in any one of claims 1-5 when executed by the processor.

Images