Model training method and image processing method and related devices, equipment, and medium

Abstract

The present application discloses a model training method and an image processing method and related devices, equipment, and media, wherein the model training method includes: outlining a first sample image using a first outlining model to obtain a first sample feature map output by a first target network layer in the first outlining model, and outlining a second sample image using a second outlining model to obtain a second sample feature map output by a second target network layer in the second outlining model; analyzing a sample image pair using a registration model to obtain a sample deformation parameter; wherein the sample image pair includes a first sample image and a second sample image; deforming the first sample feature map based on the sample deformation parameter to obtain a deformed sample feature map; and adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map. The above scheme can improve the image registration effect.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method and an image processing method and related devices, equipment, and media. Background Art

[0002] Image registration generally refers to the alignment of images of the same target in space, which is widely used in many scenarios such as face comparison and surgical planning. In addition, thanks to the rapid development of artificial intelligence technology in recent years, the use of neural networks to achieve image registration has gradually become one of the mainstream technologies for image registration.

[0003] However, the existing methods mainly constrain model training by the registration effect of the deformation parameters output by the registration model on the sample image, and the registration effect is often not ideal. In view of this, how to improve the image registration effect has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a model training method and an image processing method and related devices, equipment, and media.

[0005] The first aspect of the present application provides a model training method, including: using a first delineation model to delineate a first sample image to obtain a first sample feature map output by a first target network layer in the first delineation model, and using a second delineation model to delineate a second sample image to obtain a second sample feature map output by a second target network layer in the second delineation model; using a registration model to analyze a sample image pair to obtain a sample deformation parameter; wherein the sample image pair includes a first sample image and a second sample image; based on the sample deformation parameters, deforming the first sample feature map to obtain a deformed sample feature map; and adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map.

[0006] Therefore, the first sample image is outlined by using the first delineation model to obtain the first sample feature map output by the first target network layer in the first delineation model, and the second sample image is outlined by using the second delineation model to obtain the second sample feature map output by the second target network layer in the second delineation model. On this basis, the first sample image and the second sample image can be analyzed by using the registration model to obtain the sample deformation parameters, and the first sample feature map is deformed by using the sample deformation parameters to obtain the deformed sample feature map, and then the network parameters of the registration model are adjusted based on the difference between the deformed sample feature map and the second sample feature map. On the one hand, during the training process, the sample image is outlined by using the delineation model, which is conducive to improving the registration model's attention to the registration of target objects in different images. On the other hand, the registration model is constrained by the registration effect of the sample deformation parameters on the sample feature map, that is, the registration model can be constrained at the feature level, which is conducive to improving the performance of the registration model, thereby improving the image registration effect.

[0007] Among them, the model training method also includes: selecting several network layers in the first delineation model as the first target network layer, and selecting several network layers in the second delineation model as the second target network layer; selecting the first target network layer and the second target network layer that meet the preset conditions to combine to obtain several network layer combinations; using the sample deformation parameter to deform the first sample feature map to obtain a deformed sample feature map, including: for each network layer combination, using the sample deformation parameter to deform the first sample feature map output by the first target network layer in the network layer combination to obtain a deformed sample feature map corresponding to the network layer combination; based on the difference between the deformed sample feature map and the second sample feature map, adjusting the network parameters of the registration model, including: based on the difference between the second sample feature map output by the second target network layer in each network layer combination and the reference sample feature map, adjusting the network parameters of the registration model; wherein the reference sample feature map is the deformed sample feature map corresponding to the network layer combination.

[0008] Therefore, several network layers in the first delineation model are selected as the first target network layer, and several network layers in the second delineation model are selected as the second target network layer, and the first target network layer and the second target network layer that meet the preset conditions are selected to be combined to obtain several network layer combinations. For each network layer combination, the first sample feature map output by the first target network layer in the sample deformation parameter is used to deform the first sample feature map to obtain a deformed sample feature map corresponding to the network layer combination, so that based on the difference between the second sample feature map output by the second target network layer in each network layer combination and the reference sample feature map, the network parameters of the registration model are adjusted, and the reference sample feature map is not the deformed sample feature map corresponding to the network layer combination, so the registration effect of the sample feature maps output by the sample deformation parameters on the sample feature maps of several network layers can be used to jointly constrain the training of the registration model, which is beneficial to improving the performance of the registration model.

[0009] Among them, selecting several network layers in the first outlining model as the first target network layer respectively includes: in the first outlining model, selecting the network layer whose number of network layers is greater than a first threshold as the first target network layer; and selecting several network layers in the second outlining model as the second target network layer respectively, including: in the second outlining model, selecting the network layer whose number of network layers is greater than a second threshold as the second target network layer.

[0010] Therefore, in the first delineation model, the network layer whose number of network layers is greater than the first threshold is selected as the first target network layer, and in the second delineation model, the network layer whose number of network layers is greater than the second threshold is selected as the second target network layer. That is, the training of the registration model can be constrained by the registration effect of the sample deformation parameters on the deep features, so that the registration model can focus on learning the feature information related to the registration in the deep model, which is beneficial to improving the performance of the registration model.

[0011] Among them, the preset conditions include: the first sample feature map output by the first target network layer and the second sample feature map output by the second target network layer have the same resolution.

[0012] Therefore, the first sample feature map output by the first target network layer is set to have the same resolution as the second sample feature map output by the second target network layer. Therefore, the model training can be constrained without other image processing during the training process, which is conducive to improving the convenience of model training.

[0013] Wherein, both the first sample image and the second sample image contain a target object, the first delineation model delineates a first delineation result of the target object in the first sample image, the second delineation model delineates a second delineation result of the target object in the second sample image, and the delineation result represents a predicted contour of the target object; before adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map, the method also includes: deforming the first delineation result using the sample deformation parameters to obtain a deformed delineation result, and obtaining a first loss based on the difference between the deformed delineation result and the second delineation result; adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map, including: obtaining a second loss based on the difference between the deformed sample feature map and the second sample feature map; obtaining a total loss based on the first loss and the second loss; and adjusting the network parameters of the registration model, the first delineation model, and the second delineation model based on the total loss.

[0014] Therefore, the first delineation model delineates a first delineation result of the target object in the first sample image, and the second delineation model delineates a second delineation result of the target object in the second sample image, and the delineation result represents the predicted contour of the target object. On this basis, the first delineation result is deformed by using the sample deformation parameter to obtain a deformed delineation result, and a first loss is obtained based on the difference between the deformed structure result and the second delineation result, and a second loss is obtained based on the difference between the deformed sample feature map and the second sample feature map. The total loss is obtained based on the first loss and the second loss, and then the network parameters of the registration model, the first delineation model and the second delineation model are adjusted based on the total loss. Therefore, on the one hand, the registration model and the first delineation model and the second delineation model can be trained collaboratively, so that the registration model and the first delineation model and the second delineation model promote and complement each other, which is conducive to jointly improving the performance of the registration model, the first delineation model and the second delineation model. On the other hand, the registration effect of the delineation result by the sample deformation parameter can constrain the registration model to focus on the registration target object, which is conducive to meeting the use scenario of focusing on the registration effect of the target object.

[0015] Among them, the first sample image has a first labeling result, the second sample image has a second labeling result, and the labeling result represents the sample contour of the target object; before obtaining the total loss based on the first loss and the second loss, the method also includes: obtaining a third loss based on the difference between the first labeling result and the first delineation result, and obtaining a fourth loss based on the difference between the second labeling result and the second delineation result; deforming the first sample image using the sample deformation parameter to obtain a deformed sample image, and obtaining a fifth loss based on the difference between the deformed sample image and the second sample image; obtaining a total loss based on the first loss and the second loss, including: obtaining a total loss based on the first loss, the second loss, the third loss, the fourth loss and the fifth loss.

[0016] Therefore, the first sample image has a first labeling result, the second sample image has a second labeling result, and the labeling result represents the sample contour of the target object. On this basis, based on the difference between the first labeling result and the first outlining result, a third loss is obtained, and based on the difference between the second labeling result and the second outlining result, a fourth loss is obtained, and the first sample image is deformed using the sample deformation parameters to obtain a deformed sample image, and based on the difference between the deformed sample image and the second sample image, a fifth loss is obtained, so that based on the first loss, the second loss, the third loss, the fourth loss and the fifth loss, the total loss is obtained, and then the outlining performance of the outlining model can be further improved by constraining the outlining results and the labeling results, and the registration performance of the registration model can be improved by the registration effect of the sample deformation parameters on the sample image.

[0017] Wherein, when the first sample image and the second sample image are of the same modality, the first delineation model and the second delineation model have the same network structure; or, when the first sample image and the second sample image are of different modalities, the first delineation model and the second delineation model have different network structures.

[0018] Therefore, when the first sample image and the second sample image are of the same modality, the first outlining model and the second outlining model have the same network structure, which is conducive to reducing the complexity of training; and when the first sample image and the second sample image are of different modalities, the first outlining model and the second outlining model have different network structures, which can be conducive to improving the performance of the registration model for registering images of different modalities.

[0019] The second aspect of the present application provides an image processing method, including: acquiring a first image and a second image to be registered; analyzing the first image and the second image using a registration model to obtain deformation parameters; wherein the registration model is obtained using the model training method in the first aspect above.

[0020] The third aspect of the present application provides a model training device, including: a first outlining module, a second outlining module, an analysis module, a deformation module and an adjustment module, the first outlining module is used to use the first outlining model to outline the first sample image to obtain the first sample feature map output by the first target network layer in the first outlining model, the second outlining module is used to use the second outlining model to outline the second sample image to obtain the second sample feature map output by the second target network layer in the second outlining model; the analysis module is used to analyze the sample image pair using the registration model to obtain sample deformation parameters; wherein the sample image pair includes the first sample image and the second sample image; the deformation module is used to deform the first sample feature map based on the sample deformation parameters to obtain a deformed sample feature map; the adjustment module is used to adjust the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map.

[0021] The fourth aspect of the present application provides an image processing device, including: an acquisition module, an analysis module and a registration module, the acquisition module is used to acquire a first image and a second image to be registered; the analysis module is used to analyze the first image and the second image using a registration model to obtain deformation parameters; wherein the registration model is obtained using the model training device in the third aspect above; the registration module is used to register the first image and the second image based on the deformation parameters.

[0022] The fifth aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory to implement the model training method in the first aspect above.

[0023] The sixth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the model training method in the first aspect described above is implemented.

[0024] In the above scheme, the first sample image is outlined by the first delineation model to obtain the first sample feature map output by the first target network layer in the first delineation model, and the second sample image is outlined by the second delineation model to obtain the second sample feature map output by the second target network layer in the second delineation model. On this basis, the first sample image and the second sample image can be analyzed by the registration model to obtain the sample deformation parameters, and the first sample feature map is deformed by the sample deformation parameters to obtain the deformed sample feature map, and then the network parameters of the registration model are adjusted based on the difference between the deformed sample feature map and the second sample feature map. On the one hand, during the training process, the sample image is outlined by the delineation model, which is conducive to improving the registration model's attention to the registration of target objects in different images. On the other hand, the registration model is constrained by the registration effect of the sample deformation parameters on the sample feature map, that is, the registration model can be constrained at the feature level, which is conducive to improving the performance of the registration model, thereby improving the image registration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of an embodiment of the model training method of the present application;

[0026] Figure 2 It is a flowchart of another embodiment of the model training method of the present application;

[0027] Figure 3 It is a process diagram of an embodiment of the model training method of the present application;

[0028] Figure 4 It is a flowchart of an embodiment of the image processing method of the present application;

[0029] Figure 5 It is a schematic diagram of the framework of an embodiment of the model training device of the present application;

[0030] Figure 6 It is a schematic diagram of the framework of an embodiment of the image processing device of the present application;

[0031] Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0032] Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0033] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0034] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0035] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0036] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the model training method of the present application. Specifically, it may include the following steps:

[0037] Step S11: outlining the first sample image using the first outlining model to obtain a first sample feature map output by the first target network layer in the first outlining model.

[0038] Step S12: Use the second delineation model to delineate the second sample image to obtain a second sample feature map output by the second target network layer in the second delineation model.

[0039] In one implementation scenario, the first sample image and the second sample image can be regarded as a sample image pair. It should be noted that in the registration process of two images, one of the images is usually required to be used as the registration image and the other image as the registered image, and the registered image does not undergo any rigid or non-rigid transformation during the registration process, while the registration image needs to undergo rigid or non-rigid transformation. Exemplarily, the first sample image can be used as the registration image, and the second sample image can be used as the registered image; or, without loss of generality, the first sample image can be used as the registered image, and the second sample image can be used as the registration image, which is not limited here.

[0040] In an implementation scenario, the first sample image and the second sample image may both contain a target object, and the sample images may be set according to actual application needs. For example, in a surgical planning application scenario, the target object may include, but is not limited to, tumors, hematomas, cysts, etc., and accordingly, the sample images may include, but are not limited to, CT (Computed Tomography) images, MR (Magnetic Resonance) images, etc., which are not limited here; or, in a face comparison application scenario, the target object may include a face, and accordingly, the sample images may include, but are not limited to, portraits taken by surveillance cameras, mobile phones, etc., portraits drawn by hand-drawing, computer drawing, etc., which are not limited here. Other application scenarios can be deduced by analogy, and no examples are given one by one here.

[0041] In an implementation scenario, when the first sample image and the second sample image are of the same modality, the first delineation model and the second delineation model may have the same network structure. It should be noted that if the images are acquired by the same method, the two modalities can be considered to be the same, otherwise the two modalities can be considered to be different. Taking the surgical planning application scenario as an example, the first sample image and the second sample image can be obtained by scanning the patient with the same CT with the same parameters, then the first sample image and the second sample image can be considered to be of the same modality; or, taking the face comparison application scenario as an example, the first sample image and the second sample image can be taken by the same surveillance camera, then the first sample image and the second sample image can be considered to be of the same modality. Other application scenarios can be deduced by analogy, and examples are not given one by one here.

[0042] In one implementation scenario, when the first sample image and the second sample image are of different modalities, the first delineation model and the second delineation model have different network structures. It should be noted that the specific meaning of the modality can be found in the aforementioned related description, which will not be repeated here. Taking the surgical planning application scenario as an example, the first sample image and the second sample image can be obtained by CT and MR scanning of the patient respectively, then the first sample image and the second sample image can be considered to be of different modalities; or, taking the face comparison application scenario as an example, the first sample image can be obtained by taking a surveillance camera, and the second sample image can be obtained by drawing a computer, then the first sample image and the second sample image can be considered to be of different modalities. Other application scenarios can be deduced by analogy, and examples will not be given one by one here.

[0043] In an implementation scenario, as mentioned above, according to application needs (such as whether the aforementioned first sample image and the second sample image are of the same modality), the first outlining model and the second outlining model can be set to have the same or different network structures. In the case where the first outlining model and the second outlining model have the same network structure, the first outlining model and the second outlining model can be the same network model or two network models. That is to say, two independent network models can be used to outline the first sample image and the second sample image, respectively, or the first sample image and the second sample image can also share the same network model. Specifically, the first outlining model and the second outlining model can be network models with an encoder-decoder (i.e., encoder-decoder) structure such as U-Net. For example, the first outlining model and the second outlining model can adopt a U-Net model with a completely consistent network structure. When the first outlining model and the second outlining model have different network structures, the first outlining model and the second outlining model may also adopt similar network structures, such as the first outlining model and the second outlining model may adopt U-Net models with different numbers of network layers, that is, the first outlining model and the second outlining model both have encoder-decoder structures, but the number of network layers may be different, such as the encoder and decoder of the first outlining model have an M-layer structure, while the encoder and decoder of the second outlining model have an N-layer structure, and M is not equal to N; in addition, when the first outlining model and the second outlining model have different network structures, the first outlining model and the second outlining model may also adopt completely different network structures, such as the first outlining model may be a network model with an encoder-decoder structure such as U-Net, and the second outlining model may be a network model without an encoder-decoder structure such as CRFasRNN.

[0044] In an implementation scenario, several network layers in the first delineation model can be selected as the first target network layer, and several network layers in the second delineation model can be selected as the second target network layer, and the first target network layer and the second target network layer that meet the preset conditions can be selected to combine to obtain several network layer combinations. On this basis, for each group of network layer combinations, the first delineation model can be used to delineate the first sample image to obtain the first sample feature map output by the first target network layer in the first delineation model, and the second sample image can be delineated using the second delineation model to obtain the second sample feature map output by the second target network layer in the second delineation model. It should be noted that the number of first target network layers selected in the first delineation model and the number of second target network layers selected in the second delineation model can be the same. For example, one network layer can be selected in the first outlining model as the first target network layer, and one network layer can be selected in the second outlining model as the second target network layer; or, more generally, L network layers can be selected in the first outlining model as the first target network layer, and L network layers can be selected in the second outlining model as the second target network layer, where the specific value of L can be set to 1, 2, 3, etc., and is not limited here.

[0045] In a specific implementation scenario, the preset conditions may include: the first sample feature map output by the first target network layer and the second sample feature map output by the second target network layer have the same resolution. For example, taking the example of selecting two network layers in the first delineation model and the second delineation model as the first target network layer and the second target network layer, for the convenience of description, the two first target network layers selected in the first delineation model can be respectively recorded as p1 and p2, and the two second target network layers selected in the second delineation model can be respectively recorded as q1 and q2, wherein the resolutions of the first sample feature map output by the first target network layer p1 and the first target network layer p2 are R1 and R2, respectively, and the resolutions of the second sample feature map output by the second target network layer q1 and the second target network layer q2 are R1 and R2, respectively. In this case, the first target network layer p1 and the second target network layer q1 can be combined as a network layer combination, and the first target network layer p2 and the second target network layer q2 can be combined as a network layer combination. Other situations can be deduced by analogy, and no examples are given here one by one. In the above method, the first sample feature map output by the first target network layer is set to have the same resolution as the second sample feature map output by the second target network layer. Therefore, the model training can be constrained without other image processing during the training process, which is conducive to improving the convenience of model training.

[0046] In a specific implementation scenario, in the first outlining model, a network layer whose number of network layers is greater than the first threshold can be specifically selected as the second target network layer, and in the second outlining model, a network layer whose number of network layers is greater than the second threshold can be specifically selected as the second target network layer. The first threshold and the second threshold can be set according to actual conditions. For example, when the outlining model is relatively deep, the threshold can be set larger, and when the outlining model is relatively shallow, the threshold can be set smaller. Taking the outlining model including S network layers as an example, the target network layer can be selected from the last t% of the network layers in the S network layers. Taking S as 50 and t as 20 as an example, the target network layer can be selected from the last 10 network layers in the outlining model. Other situations can be deduced by analogy, and examples are not given one by one here. In the above method, a network layer whose number of network layers is greater than a first threshold is selected in the first delineation model as the first target network layer, and a network layer whose number of network layers is greater than a second threshold is selected in the second delineation model as the second target network layer. That is, the training of the registration model can be constrained by the registration effect of the sample deformation parameters on the deep features, so that the registration model can focus on learning the feature information related to the registration in the deep model, which is beneficial to improving the performance of the registration model.

[0047] It should be noted that the first outlining model and the second outlining model can be pre-trained. In this case, the "model" referred to in the disclosed embodiment of the model training method of the present application specifically refers to the registration model; in addition, in order to further improve the model performance of the registration model, the first outlining model and the second outlining model can also be trained collaboratively with the registration model, that is, in the training process of the registration model, the first outlining model and the second outlining model can also be trained together. In this case, the "model" referred to in the disclosed embodiment of the model training method of the present application can specifically include the first outlining model, the second outlining model and the registration model. The specific process of collaborative training can refer to the following disclosed embodiments, which will not be repeated here.

[0048] Step S13: Analyze the sample image pair using the registration model to obtain sample deformation parameters.

[0049] Specifically, the sample image pair can be input into the registration model, and the registration model can be processed and analyzed to obtain the sample deformation parameters. The registration model can specifically include but is not limited to convolutional neural networks, etc., which are not limited here. In addition, it should be noted that the sample deformation parameters can be represented by a multidimensional tensor, which can specifically include the position mapping relationship between corresponding pixels in the first sample image and the second sample image, and the corresponding pixels specifically refer to pixels representing the same position. Taking the surgical planning application scenario as an example, the pixel point P1 in the first sample image represents the apex of the liver, and the pixel point Q1 in the second sample image represents the apex of the liver, then the pixel point P1 and the pixel point Q1 can be considered as corresponding pixels; or, taking the face comparison application scenario as an example, the pixel point P2 in the first sample image represents the corner of the eye, and the pixel point Q2 in the second sample image represents the corner of the eye, then the pixel point P2 and the pixel point Q2 can be considered as corresponding pixels. Other application scenarios can be deduced by analogy, and no examples are given here one by one.

[0050] Step S14: deforming the first sample feature map based on the sample deformation parameter to obtain a deformed sample feature map.

[0051] Specifically, each pixel point in the first sample feature map may be transformed using the sample deformation parameters to change the position of each pixel point in the first sample feature map, thereby obtaining a deformed sample feature map.

[0052] In an implementation scenario, as mentioned above, several network layers can be selected in the first delineation model and the second delineation model respectively, so as to obtain several network layer combinations. For details, please refer to the above-mentioned related descriptions, which will not be repeated here. On this basis, for each network layer combination, the first sample feature map output by the first target network layer in the network layer combination can be deformed using the sample deformation parameters to obtain a deformed sample feature map corresponding to the network layer combination. Still taking the example of selecting two network layers as the first target network layer and the second target network layer in the first delineation model and the second delineation model respectively, for the network layer combination composed of the first target network layer p1 and the second target network layer q1, the first sample feature map output by the first target network layer p1 can be deformed using the sample deformation parameters to obtain the deformed sample feature map corresponding to the network layer combination; similarly, for the network layer combination composed of the first target network layer p2 and the second target network layer q2, the first sample feature map output by the first target network layer p2 can be deformed using the sample deformation parameters to obtain the deformed sample feature map corresponding to the network layer combination. The same can be said for other situations, which will not be listed here one by one.

[0053] Step S15: Based on the difference between the deformed sample feature map and the second sample feature map, adjust the network parameters of the registration model.

[0054] Specifically, a loss value may be obtained based on the difference between the deformed sample feature map and the second sample feature map, and the network parameters of the registration model may be adjusted based on the loss value.

[0055] In one implementation scenario, the difference between the deformed sample feature map and the second sample feature map can be measured by a loss function such as the L2 norm to obtain a loss value. For the specific calculation process, please refer to the technical details of the loss function such as the L2 norm, which will not be repeated here.

[0056] In an implementation scenario, an optimization method such as gradient descent can be used to adjust the network parameters of the registration model. For the specific adjustment process, please refer to the technical details of the optimization method such as gradient descent, which will not be described here.

[0057] In an implementation scenario, as mentioned above, several network layers can be selected in the first delineation model and the second delineation model respectively, so as to obtain several network layer combinations. For details, please refer to the above-mentioned related descriptions, which will not be repeated here. On this basis, the network parameters of the registration model can be adjusted based on the difference between the second sample feature map output by the second target network layer in each network layer combination and the reference sample feature map, and the reference sample feature map is the deformed sample feature map corresponding to the network layer combination. The above method can comprehensively adjust the network parameters of the registration model according to the registration effects corresponding to each network layer combination, which is beneficial to improving the model performance of the registration model.

[0058] In a specific implementation scenario, for each network layer combination, the registration loss corresponding to the network layer combination can be obtained based on the difference between the second sample feature map output by the second target network layer in the network layer combination and the reference sample feature map, so that the registration loss corresponding to each network layer combination can be weighted by using the reference weights corresponding to each network layer combination to obtain the weighted loss, and then the network parameters of the registration model can be adjusted based on the weighted loss. The reference weights corresponding to each network layer combination can be set according to the actual application needs. For example, when each network layer combination is equally important in the training process of the constrained registration model, the reference weights corresponding to each network layer combination can be set to the same (e.g., all set to 1); or, when the deeper the network depth of the network layer combination, the greater its reference value, the reference weights corresponding to each network layer combination can be positively correlated with the network depth of the network layer combination; or, similarly, when the deeper the network depth of the network layer combination, the smaller its reference value, the reference weights corresponding to each network layer combination can be negatively correlated with the network depth of the network layer combination. Other situations can be deduced by analogy, and examples are not given one by one here.

[0059] In a specific implementation scenario, for the sake of ease of description, we still take the example of selecting two network layers in the first delineation model and the second delineation model as the first target network layer and the second target network layer respectively. For the network layer combination composed of the first target network layer p1 and the second target network layer q1, a loss value can be obtained based on the difference between the second sample feature map output by the second target network layer q1 and the deformed sample feature map corresponding to the network layer combination; similarly, for the network layer combination composed of the first target network layer p2 and the second target network layer q2, a loss value can be obtained by using the difference between the first sample feature map output by the second target network layer q2 and the deformed sample feature map corresponding to the network layer combination. On this basis, a total loss value can be obtained based on these two loss values, so that the network parameters of the registration model can be adjusted based on the total loss value. Other situations can be deduced by analogy, and examples are not given one by one here.

[0060] In the above scheme, a sample image pair of the target object is obtained, and the sample image pair includes a first sample image and a second sample image. The first sample image is delineated by using a first delineation model to obtain a first sample feature map output by the first target network layer in the first delineation model, and the second sample image is delineated by using a second delineation model to obtain a second sample feature map output by the second target network layer in the second delineation model. On this basis, the model can be configured to analyze the first sample image and the second sample image to obtain sample deformation parameters, and the first sample feature map is deformed by using the sample deformation parameters to obtain a deformed sample feature map, and then the network parameters of the registration model are adjusted based on the difference between the deformed sample feature map and the second sample feature map. On the one hand, during the training process, the delineation model is used to delineate the sample image, which is conducive to improving the registration model's attention to the registration of target objects in different images. On the other hand, the registration model is constrained by the registration effect of the sample deformation parameters on the sample feature map, that is, the registration model can be constrained at the feature level, which is conducive to improving the performance of the registration model, thereby improving the image registration effect.

[0061] See also Figure 2 , Figure 2 This is a flowchart of another embodiment of the model training of the present application. Specifically, it may include the following steps:

[0062] Step S21: outlining the first sample image using the first outlining model to obtain a first sample feature map output by the first target network layer in the first outlining model and a first outlining result of the target object in the first sample image.

[0063] Step S22: Use the second delineation model to delineate the second sample image to obtain a second sample feature map output by the second target network layer in the second delineation model and a second delineation result of the target object in the second sample image.

[0064] In the disclosed embodiment, as mentioned above, the first sample image and the second sample image may both contain the target object, and the delineation result represents the predicted contour of the target object, that is, the first delineation result represents the predicted contour of the target object in the first sample image, and the second delineation result represents the predicted contour of the target object in the second sample image. Specifically, the delineation result may include whether each sample pixel in the sample image belongs to the target object, that is, the first delineation result may include whether each sample pixel in the first sample image belongs to the target object, and the second delineation result may include whether each sample pixel in the second sample image belongs to the target object. In addition, for the first delineation model and the second delineation model, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0065] In one implementation scenario, in order to facilitate subsequent loss calculation, a mask image (mask) can be used to represent the delineation result, that is, a first pixel value (e.g., 255) can be used to represent that the sample pixel belongs to the target object, and a second pixel value (e.g., 0) can be used to represent that the sample pixel does not belong to the target object. In this case, a first predicted mask image can be used to represent the first delineation result, and the pixel value of each sample pixel in the first predicted mask image is set according to whether the corresponding pixel in the first sample image belongs to the target object predicted by the first delineation model. Similarly, a second predicted mask image can be used to represent the second delineation result, and the pixel value of each sample pixel in the second predicted mask image is set according to whether the corresponding pixel in the second sample image belongs to the target object predicted by the second delineation model. It should be noted that the corresponding pixel represents the sample pixel in the sample image that has the same position as the sample pixel in the mask image. For example, the sample pixel at (u, v) in the mask image corresponds to a pixel in the sample image, which can be a sample pixel at (u, v) in the sample image.

[0066] Step S23: Analyze the sample image pair using the registration model to obtain sample deformation parameters.

[0067] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0068] Step S24: deforming the first sample feature map based on the sample deformation parameter to obtain a deformed sample feature map.

[0069] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0070] Step S25: deforming the first delineation result using the sample deformation parameter to obtain a deformed delineation result, and obtaining a first loss based on the difference between the deformed delineation result and the second delineation result.

[0071] In one implementation scenario, as mentioned above, in order to facilitate loss calculation, a mask image can be used to represent the delineation result. In this case, the first predicted mask image can be deformed using the sample deformation parameter to obtain a deformed mask image, and based on the difference between the deformed mask image and the second predicted mask image, the first loss is obtained. Specifically, a loss function such as dice loss can be used to measure the difference between the deformed mask image and the second predicted mask image to obtain the second loss. For the specific calculation process, please refer to the technical details of loss functions such as dice loss, which will not be repeated here.

[0072] Step S26: Obtain a second loss based on the difference between the deformed sample feature map and the second sample feature map.

[0073] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0074] Step S27: Based on the first loss and the second loss, the total loss is obtained.

[0075] In one implementation scenario, the first loss and the second loss can be weighted to obtain a total loss. Specifically, the reference weights of the first loss and the second loss can be set according to their importance in the constraint model training process. For example, when the feature map is more important, the reference weight of the second loss can be set to be greater than the reference weight of the first loss; or, when the delineation result is more important, the reference weight of the first loss can be set to be greater than the reference weight of the second loss; or, when the two are equally important, the reference weight of the first loss can be set to be equal to the reference weight of the second loss. No limitation is made here.

[0076] In one implementation scenario, the first sample image may have a first labeling result, the second sample image may have a second labeling result, and the labeling result represents a sample outline of the target object. On this basis, a third loss may be further obtained based on the difference between the first labeling result and the first delineation result, and a fourth loss may be further obtained based on the difference between the second labeling result and the second delineation result.

[0077] In a specific implementation scenario, similar to the delineation result, in order to facilitate the subsequent loss calculation, a mask image (mask) can be used to represent the labeling result, that is, a first pixel value (such as 255) can be used to represent that the sample pixel belongs to the target object, and a second pixel value (such as 0) can be used to represent that the sample pixel does not belong to the target object. In this case, the first sample mask image can be used to represent the first labeling result, and the pixel value of each sample pixel in the first sample mask image is set according to whether the corresponding pixel in the first sample image belongs to the target object. Similarly, the second sample mask image can be used to represent the second labeling result, and the pixel value of each sample pixel in the second sample mask image is set according to whether the corresponding pixel in the second sample image belongs to the target object. It should be noted that the corresponding pixel represents the sample pixel in the sample image that has the same position as the sample pixel in the mask image. For example, the sample pixel at (u, v) in the mask image corresponds to a pixel in the sample image, which can specifically be a sample pixel at (u, v) in the sample image. On this basis, a third loss can be obtained based on the difference between the first sample mask image and the first predicted mask image, and a fourth loss can be obtained based on the difference between the second sample mask image and the second predicted mask image.

[0078] In a specific implementation scenario, a loss function such as cross entropy loss or dice loss may be used to measure the difference between the first labeling result and the first delineation result, and the difference between the second labeling result and the second delineation result. For the specific calculation process, please refer to the technical details of the loss function such as cross entropy loss or dice loss, which will not be described here.

[0079] In one implementation scenario, the first sample image can be deformed using the sample deformation parameter to obtain a deformed sample image, and based on the difference between the deformed sample image and the second sample image, a fifth loss can be further obtained. Specifically, the specific process of the deformation operation can refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here. In addition, a loss function such as the L2 norm can be used to measure the difference between the deformed sample image and the second sample image. For the specific calculation process, the technical details of the loss function such as the L2 norm can be referred to, which will not be repeated here.

[0080] In one implementation scenario, the sixth loss can be further obtained based on the smoothness of the sample deformation parameter. It should be noted that the smoothness can be measured by calculating the gradient of the sample deformation parameter, and the sixth loss can be obtained based on the gradient calculation, and the larger the gradient, the larger the sixth loss.

[0081] In an implementation scenario, please refer to Figure 3 , Figure 3This is a schematic diagram of the process of an embodiment of the model training method of the present application. Figure 3 As shown, the first loss, the second loss, the third loss, the fourth loss, the fifth loss and the sixth loss can be calculated respectively in the above manner, and the total loss can be obtained based on the above six losses. For example, the reference weights of the above six losses can be obtained according to the importance of the above six losses, and the above six losses can be weighted respectively by using the reference weights of the above six losses to obtain the total loss.

[0082] Step S28: Based on the total loss, adjust the network parameters of the registration model, the first delineation model and the second delineation model.

[0083] Specifically, an optimization method such as gradient descent can be used to adjust the network parameters of the registration model, the first delineation model, and the second delineation model based on the total loss. For the specific adjustment process, please refer to the technical details of the optimization method such as gradient descent, which will not be repeated here.

[0084] It should be noted that after the training is completed, the image can be registered using the registration model obtained through training. For details, please refer to the following public embodiments, which will not be described in detail here. In addition, the image can also be delineated using the first delineation model or the second delineation model obtained through training. Specifically, the first delineation model can be used to delineate an image of a first modality, where the first modality is the modality of a first sample image (e.g., a CT image), or the second delineation model can be used to delineate an image of a second modality, where the second modality is the modality of a second sample image (e.g., an MR image).

[0085] In the above scheme, the first delineation model delineates a first delineation result of the target object in the first sample image, and the second delineation model delineates a second delineation result of the target object in the second sample image, and the delineation result represents the predicted contour of the target object. On this basis, the first delineation result is deformed by using the sample deformation parameter to obtain a deformed delineation result, and a first loss is obtained based on the difference between the deformed structure result and the second delineation result, and a second loss is obtained based on the difference between the deformed sample feature map and the second sample feature map. Based on the first loss and the second loss, a total loss is obtained, and then based on the total loss, the network parameters of the registration model, the first delineation model and the second delineation model are adjusted. Therefore, on the one hand, the registration model and the first delineation model and the second delineation model can be trained collaboratively, so that the registration model and the first delineation model and the second delineation model promote and complement each other, which is conducive to jointly improving the performance of the registration model, the first delineation model and the second delineation model. On the other hand, the registration effect of the delineation result by the sample deformation parameter can constrain the registration model to focus on the registration target object, which is conducive to meeting the use scenario of focusing on the registration effect of the target object.

[0086] See also Figure 4 , Figure 4 This is a flow chart of an embodiment of the image processing method of the present application. Specifically, it may include the following steps:

[0087] Step S41: Acquire the first image and the second image to be registered.

[0088] In the embodiments of the present disclosure, it should be noted that, in the process of registering two images, one of the images is usually required to be used as the registering image and the other image as the registered image, and the registered image does not undergo any rigid or non-rigid transformation during the registration process, while the registering image needs to undergo rigid or non-rigid transformation. In addition, in the embodiments of the present disclosure, for example, the first image can be used as the registering image and the second image can be used as the registered image; or, without loss of generality, the first image can be used as the registered image and the second image can be used as the registering image, which is not limited here.

[0089] In an implementation scenario, similar to the sample image, the first image and the second image can also be set according to actual application needs. For example, in the surgical planning application scenario, the first image and the second image can include but are not limited to: CT (Computed Tomography) images, MR (Magnetic Resonance) images, etc., which are not limited here; or, in the face comparison application scenario, the first image and the second image can include but are not limited to: portraits taken by surveillance cameras, mobile phones, etc., portraits drawn by hand-drawing, computer drawing, etc., which are not limited here. Other application scenarios can be deduced by analogy, and no examples are given here one by one.

[0090] In an implementation scenario, as described above, in the process of training the registration model, the network structure of the first delineation model and the second delineation model can be set according to whether the first sample image and the second sample image are of the same modality. Similarly, in the application stage, the first image and the second image of the same modality can be obtained, or the first image and the second image of different modalities can be obtained according to actual application needs, which is not limited here.

[0091] In a specific implementation scenario, taking the surgical planning application scenario as an example, a CT image can be used as the first image, and an MR image or a PET-CT image can be used as the second image to achieve registration between medical images of different modalities, so that the target object (such as a tumor, cyst, hematoma, etc.) can be better observed by combining multi-modal images; or, taking the face comparison application scenario as an example, a face image captured by a surveillance camera can be used as the first image, and a computer or hand-drawn portrait image can be used as the second image, so that the fit between the two can be observed. Other application scenarios can be deduced in the same way, and no examples are given here one by one.

[0092] In a specific implementation scenario, taking the surgical planning application scenario as an example, the CT images of the patient at different time nodes can be used as the first image and the second image respectively to achieve registration between the medical images at different time nodes, so that the changing trend of the patient's health status at different time nodes can be observed (such as the tumor tends to become smaller, the cyst tends to become smaller, etc.), and then in the surgical planning process, the original surgical plan can be adjusted directly without having to start planning from scratch. Other application scenarios can be deduced by analogy, and no examples are given here one by one.

[0093] Step S42: Analyze the first image and the second image using the registration model to obtain deformation parameters.

[0094] In the embodiment of the present disclosure, the registration model is obtained by using the steps in any of the above-mentioned model training method embodiments. For details, please refer to the above-mentioned disclosed embodiments and will not be repeated here.

[0095] Step S43: registering the first image and the second image based on the deformation parameter.

[0096] Specifically, taking the first image as the registration image and the second image as the registered image as an example, the first image can be deformed using deformation parameters to obtain a deformed image, so as to achieve registration between the first image and the second image.

[0097] In one implementation scenario, the deformed image and the second image may be superimposed, so that the target object (eg, a tumor, a cyst, a hematoma, etc.) may be better observed by combining the first image and the second image.

[0098] In one implementation scenario, the deformed image and the second image may also be displayed simultaneously (eg, displayed side by side) so that the user can observe the deformed image and the second image simultaneously.

[0099] It should be noted that the above two methods can be selected according to needs during actual application, that is, the first superimposed display method or the second simultaneous display method can be selected, or the superimposed display method can be selected first and then the simultaneous display method, or the simultaneous display method can be selected first and then the superimposed display method, and there is no limitation here.

[0100] In the above scheme, the first image and the second image to be registered are obtained, and the first image and the second image are analyzed using the registration model to obtain deformation parameters. The registration model is obtained using the steps in any of the aforementioned model training method embodiments. On this basis, the first image and the second image are registered based on the deformation parameters, which is beneficial to improving the image registration effect.

[0101] See also Figure 5 , Figure 5 : is a schematic diagram of a framework of an embodiment of a model training device 50. The model training device 50 includes: a first delineation module 51, a second delineation module 52, an analysis module 53, a deformation module 54 and an adjustment module 55. The first delineation module 52 is used to delineate the first sample image using the first delineation model to obtain the first sample feature map output by the first target network layer in the first delineation model. The second delineation module 52 is used to delineate the second sample image using the second delineation model to obtain the second sample feature map output by the second target network layer in the second delineation model; the analysis module 53 is used to analyze the sample image pair using the registration model to obtain the sample deformation parameter; wherein the sample image pair includes the first sample image and the second sample image; the deformation module 54 is used to deform the first sample feature map based on the sample deformation parameter to obtain the deformed sample feature map; the adjustment module 55 is used to adjust the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map.

[0102] The above scheme, on the one hand, uses the outlining model to outline the sample image during the training process, which is conducive to improving the registration model's attention to the registration of target objects in different images; on the other hand, it constrains the registration model through the registration effect of the sample deformation parameters on the sample feature map, that is, it can constrain the registration model at the feature level, so it is conducive to improving the performance of the registration model, thereby improving the image registration effect.

[0103] In some disclosed embodiments, the model training device 50 also includes a first selection module for selecting several network layers in the first delineation model as first target network layers, and the model training device 50 also includes a second selection module for selecting several network layers in the second delineation model as second target network layers; the model training device 50 also includes a combination module for selecting the first target network layer and the second target network layer that meet the preset conditions for combination to obtain several network layer combinations; the deformation module 54 is specifically used to deform the first sample feature map output by the first target network layer in the network layer combination for each network layer combination using the sample deformation parameter to obtain a deformed sample feature map corresponding to the network layer combination; the adjustment module 55 is specifically used to adjust the network parameters of the registration model based on the difference between the second sample feature map output by the second target network layer in each network layer combination and the reference sample feature map; wherein the reference sample feature map is the deformed sample feature map corresponding to the network layer combination.

[0104] Therefore, the registration effect of the sample feature maps output by several network layers can be used through the sample deformation parameters to jointly constrain the training of the registration model, which is beneficial to improving the performance of the registration model.

[0105] In some disclosed embodiments, the first selection module is specifically used to select a network layer whose number of network layers is greater than a first threshold in the first delineation model as the first target network layer; the second selection module is specifically used to select a network layer whose number of network layers is greater than a second threshold in the second delineation model as the second target network layer.

[0106] Therefore, in the first delineation model, the network layer whose number of network layers is greater than the first threshold is selected as the first target network layer, and in the second delineation model, the network layer whose number of network layers is greater than the second threshold is selected as the second target network layer. That is, the training of the registration model can be constrained by the registration effect of the sample deformation parameters on the deep features, so that the registration model can focus on learning the feature information related to the registration in the deep model, which is beneficial to improving the performance of the registration model.

[0107] In some disclosed embodiments, the preset condition includes: a first sample feature map output by the first target network layer and a second sample feature map output by the second target network layer have the same resolution.

[0108] Therefore, the first sample feature map output by the first target network layer is set to have the same resolution as the second sample feature map output by the second target network layer. Therefore, the model training can be constrained without other image processing during the training process, which is conducive to improving the convenience of model training.

[0109] In some disclosed embodiments, the first sample image and the second sample image both contain a target object, the first delineation model delineates a first delineation result of the target object in the first sample image, the second delineation model delineates a second delineation result of the target object in the second sample image, and the delineation result represents a predicted contour of the target object; the model training device 50 also includes a calculation module for deforming the first delineation result using a sample deformation parameter to obtain a deformed delineation result, and obtaining a first loss based on the difference between the deformed delineation result and the second delineation result; the adjustment module 55 includes a feature difference measurement submodule for obtaining a second loss based on the difference between the deformed sample feature map and the second sample feature map; the adjustment module 55 includes a loss calculation submodule for obtaining a total loss based on the first loss and the second loss; the adjustment module 55 includes a parameter optimization submodule for adjusting the network parameters of the registration model, the first delineation model, and the second delineation model based on the total loss.

[0110] Therefore, on the one hand, the registration model can be trained collaboratively with the first and second outlining models, so that the registration model, the first and second outlining models promote and complement each other, which is conducive to jointly improving the performance of the registration model, the first and second outlining models. On the other hand, the registration effect of the outlining results through the sample deformation parameters can constrain the registration model to focus on the registration target object, which is conducive to meeting the usage scenarios that focus on the registration effect of the target object.

[0111] In some disclosed embodiments, the first sample image has a first labeling result, the second sample image has a second labeling result, and the labeling result represents a sample contour of the target object; the adjustment module 55 also includes a delineation difference measurement submodule, which is used to obtain a third loss based on the difference between the first labeling result and the first delineation result, and to obtain a fourth loss based on the difference between the second labeling result and the second delineation result; the adjustment module 55 also includes a sample difference measurement submodule, which is used to deform the first sample image using a sample deformation parameter to obtain a deformed sample image, and to obtain a fifth loss based on the difference between the deformed sample image and the second sample image; the loss calculation submodule is specifically used to obtain a total loss based on the first loss, the second loss, the third loss, the fourth loss and the fifth loss.

[0112] Therefore, the delineation performance of the delineation model can be further improved by constraining the delineation results and the marking results, and the registration performance of the registration model can be improved by the registration effect of the sample deformation parameters on the sample image.

[0113] In some disclosed embodiments, when the first sample image and the second sample image are of the same modality, the first delineation model and the second delineation model have the same network structure; or, when the first sample image and the second sample image are of different modalities, the first delineation model and the second delineation model have different network structures.

[0114] Therefore, when the first sample image and the second sample image are of the same modality, the first outlining model and the second outlining model have the same network structure, which is conducive to reducing the complexity of training; and when the first sample image and the second sample image are of different modalities, the first outlining model and the second outlining model have different network structures, which can be conducive to improving the performance of the registration model for registering images of different modalities.

[0115] See also Figure 6 , Figure 6: is a schematic diagram of a framework of an embodiment of an image processing device 60 of the present application. The image processing device 60 includes: an acquisition module 61, an analysis module 62 and a registration module 63, wherein the acquisition module 61 is used to acquire a first image and a second image to be registered; the analysis module 62 is used to analyze the first image and the second image using a registration model to obtain deformation parameters; wherein the registration model is obtained using the model training device in any of the above-mentioned model training device embodiments; and the registration module 63 is used to register the first image and the second image based on the deformation parameters.

[0116] The above scheme obtains the first image and the second image to be registered, uses the registration model to analyze the first image and the second image to obtain deformation parameters, and the registration model is obtained by using the model training device in any of the aforementioned model training device embodiments. On this basis, the first image and the second image are registered based on the deformation parameters, which is beneficial to improving the image registration effect.

[0117] See also Figure 7 , Figure 7 : is a schematic diagram of a framework of an embodiment of an electronic device 70 of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, and the processor 72 is used to execute program instructions stored in the memory 71 to implement the steps of any of the above-mentioned model training method embodiments, or to implement the steps of any of the above-mentioned image processing method embodiments. In a specific implementation scenario, the electronic device 70 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 70 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0118] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps of any of the above-mentioned model training method embodiments, or to implement the steps of any of the above-mentioned image processing method embodiments. The processor 72 may also be referred to as a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip having signal processing capabilities. The processor 72 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 72 may be implemented by an integrated circuit chip.

[0119] The above scheme, on the one hand, uses the outlining model to outline the sample image during the training process, which is conducive to improving the registration model's attention to the registration of target objects in different images. On the other hand, the registration model is constrained by the registration effect of the sample deformation parameters on the sample feature map, that is, the registration model can be constrained at the feature level, which is conducive to improving the performance of the registration model, thereby improving the image registration effect.

[0120] See also Figure 8 , Figure 8 The schematic diagram of the framework of an embodiment of a computer-readable storage medium 80 of the present application. The computer-readable storage medium 80 stores program instructions 801 that can be executed by a processor, and the program instructions 801 are used to implement the steps of any of the above-mentioned model training method embodiments, or to implement the steps of any of the above-mentioned image processing method embodiments.

[0121] The above scheme, on the one hand, uses the outlining model to outline the sample image during the training process, which is conducive to improving the registration model's attention to the registration of target objects in different images. On the other hand, the registration model is constrained by the registration effect of the sample deformation parameters on the sample feature map, that is, the registration model can be constrained at the feature level, which is conducive to improving the performance of the registration model, thereby improving the image registration effect.

[0122] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0124] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0126] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A model training method, characterized in that: include: Delineating a first sample image using a first delineation model to obtain a first sample feature map output by a first target network layer in the first delineation model; Delineating the second sample image using the second delineation model to obtain a second sample feature map output by the second target network layer in the second delineation model; Analyzing the sample image pair using the registration model to obtain sample deformation parameters; wherein the sample image pair includes the first sample image and the second sample image; Based on the sample deformation parameter, deforming the first sample feature map to obtain a deformed sample feature map; Based on the difference between the deformed sample feature map and the second sample feature map, adjusting the network parameters of the registration model; wherein the method further comprises: Selecting several network layers in the first delineation model as the first target network layers respectively, and selecting several network layers in the second delineation model as the second target network layers respectively; Selecting the first target network layer and the second target network layer that meet a preset condition to combine, to obtain a plurality of network layer combinations; The deforming the first sample feature map based on the sample deformation parameter to obtain a deformed sample feature map includes: For each of the network layer combinations, deforming the first sample feature map output by the first target network layer in the network layer combination using the sample deformation parameter to obtain a deformed sample feature map corresponding to the network layer combination; and The adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map comprises: Based on the difference between the second sample feature map output by the second target network layer in each of the network layer combinations and the reference sample feature map, the network parameters of the registration model are adjusted; wherein the reference sample feature map is a deformed sample feature map corresponding to the network layer combination.

2. The method according to claim 1, characterized in that: The selecting a plurality of network layers in the first delineation model as the first target network layers respectively includes: In the first delineation model, selecting a network layer whose number of network layers is greater than a first threshold as the first target network layer; and, The selecting a plurality of network layers in the second delineation model as the second target network layers respectively comprises: In the second delineation model, a network layer whose number of network layers is greater than a second threshold is selected as the second target network layer.

3. The method according to claim 1, characterized in that The preset condition includes: a first sample feature map output by the first target network layer and a second sample feature map output by the second target network layer have the same resolution.

4. The method according to claim 1, characterized in that: The first sample image and the second sample image both include a target object, the first delineation model delineates a first delineation result of the target object in the first sample image, the second delineation model delineates a second delineation result of the target object in the second sample image, and the delineation results represent a predicted contour of the target object; Before adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map, the method further includes: deforming the first delineation result using the sample deformation parameter to obtain a deformed delineation result, and obtaining a first loss based on a difference between the deformed delineation result and the second delineation result; and The adjusting the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map comprises: Obtaining a second loss based on a difference between the deformed sample feature map and the second sample feature map; Obtaining a total loss based on the first loss and the second loss; Based on the total loss, network parameters of the registration model, the first delineation model, and the second delineation model are adjusted.

5. The method according to claim 4, characterized in that The first sample image has a first labeling result, the second sample image has a second labeling result, and the labeling result represents a sample contour of the target object; Before obtaining the total loss based on the first loss and the second loss, the method further includes: Based on the difference between the first labeling result and the first delineation result, a third loss is obtained, and based on the difference between the second labeling result and the second delineation result, a fourth loss is obtained; deforming the first sample image using the sample deformation parameter to obtain a deformed sample image, and obtaining a fifth loss based on a difference between the deformed sample image and the second sample image; The obtaining of a total loss based on the first loss and the second loss comprises: The total loss is obtained based on the first loss, the second loss, the third loss, the fourth loss and the fifth loss.

6. The method according to claim 1, characterized in that In a case where the first sample image and the second sample image are of the same modality, the first delineation model and the second delineation model have the same network structure; Alternatively, when the first sample image and the second sample image are of different modalities, the first delineation model and the second delineation model have different network structures.

7. An image processing method, characterized in that: include: Acquire a first image and a second image to be registered; Analyzing the first image and the second image using a registration model to obtain deformation parameters; wherein the registration model is obtained using the method according to any one of claims 1 to 6; Based on the deformation parameter, the first image and the second image are registered.

8. A model training device, characterized in that: include: A first delineation module, configured to delineate a first sample image using a first delineation model to obtain a first sample feature map output by a first target network layer in the first delineation model; A second delineation module, used to delineate the second sample image using a second delineation model to obtain a second sample feature map output by a second target network layer in the second delineation model; A first selection module is used to select a plurality of network layers in the first delineation model as the first target network layers, and select a plurality of network layers in the second delineation model as the second target network layers; A second selection module, used for selecting the first target network layer and the second target network layer that meet a preset condition for combination to obtain a plurality of network layer combinations; An analysis module, used to analyze the sample image pair using the registration model to obtain sample deformation parameters; wherein the sample image pair includes the first sample image and the second sample image; A deformation module, used to deform the first sample feature map based on the sample deformation parameter to obtain a deformed sample feature map, specifically used to deform the first sample feature map output by the first target network layer in the network layer combination using the sample deformation parameter for each of the network layer combinations to obtain a deformed sample feature map corresponding to the network layer combination; An adjustment module is used to adjust the network parameters of the registration model based on the difference between the deformed sample feature map and the second sample feature map, and is specifically used to adjust the network parameters of the registration model based on the difference between the second sample feature map output by the second target network layer in each of the network layer combinations and the reference sample feature map; wherein the reference sample feature map is the deformed sample feature map corresponding to the network layer combination.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring a first image and a second image to be registered; an analysis module, configured to analyze the first image and the second image using a registration model to obtain deformation parameters; wherein the registration model is obtained using the model training device according to claim 8; A registration module is used to register the first image and the second image based on the deformation parameter.

10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the model training method described in any one of claims 1 to 6, or to implement the image processing method described in claim 7.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the model training method described in any one of claims 1 to 6 is implemented, or the image processing method described in claim 7 is implemented.

Images