Semantics-based registration method and device, surgical robot, and electronic device

Abstract

The application discloses a semantic-based registration method and device, a surgical robot and electronic equipment. The method comprises the following steps: acquiring a first three-dimensional image and a second three-dimensional image, wherein the difference between the volume of a target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image is greater than or equal to a first threshold. Based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, the region different from the target tissue in the second three-dimensional image in the first three-dimensional image is removed to obtain a third three-dimensional image. The second three-dimensional image and the third three-dimensional image are registered by using a first model to obtain a target registration result, the first model is obtained by training based on a target loss function, the function value of the target loss function is positively correlated with a volume difference, and the volume difference indicates the difference between the volumes of the target tissues in the two three-dimensional images to be registered. Based on the method, the accuracy of the target registration result can be improved.

Description

Technical Field

[0001] This application relates to the field of medical imaging technology, and in particular to a semantic-based registration method and apparatus, a surgical robot, and an electronic device. Background Technology

[0002] In the medical field, in order to utilize three-dimensional images acquired at different times, it is necessary to register the three-dimensional images acquired at different times.

[0003] Current registration methods first identify feature points in different 3D images, and then register the different 3D images based on the feature points.

[0004] However, the registration results obtained by this method have low accuracy. Summary of the Invention

[0005] This application provides a semantic-based registration method and apparatus, surgical robot, and electronic device to improve the accuracy of registration results between three-dimensional images.

[0006] Firstly, a semantic-based registration method is provided, the method comprising: A first three-dimensional image and a second three-dimensional image are acquired. Both the first three-dimensional image and the second three-dimensional image include the target tissue. The difference between the volume of the target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image is greater than or equal to a first threshold. Based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, regions in the first three-dimensional image that are different from the target tissue in the second three-dimensional image are removed to obtain a third three-dimensional image. Using a first model, the second three-dimensional image and the third three-dimensional image are registered to obtain a target registration result. The first model is trained based on a target loss function, the function value of which is positively correlated with the volume difference. The volume difference indicates the difference in volume of the target tissue in the two three-dimensional images to be registered. The target registration result is used to align the second three-dimensional image and the third three-dimensional image.

[0007] In any embodiment of this application, before registering the second three-dimensional image and the third three-dimensional image using the first model to obtain the target registration result, the method further includes: Using the same division method, the second three-dimensional image and the third three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set; The corresponding image blocks in the first image block set and the second image block set are registered to obtain a first registration result set, and the first registration result in the first registration result set corresponds one-to-one with the image blocks in the first image block set; Cluster the first registration results in the first registration result set to obtain at least two cluster sets; Based on the at least two cluster sets, the first image patch set is divided into at least two first image patch subsets, and the second image patch set is divided into at least two second image patch subsets. The first registration results corresponding to the image patches in the first image patch subsets belong to the same cluster set, and the first registration results corresponding to the image patches in the second image patch subsets belong to another cluster set. The process of registering the second 3D image and the third 3D image using the first model to obtain the target registration result includes: Using the first model, the corresponding subsets in the at least two first image block subsets and the at least two second image block subsets are registered to obtain at least two second registration results; The target registration result is obtained based on the at least two second registration results.

[0008] In any embodiment of this application, the target tissue belongs to the target human body; Before obtaining the target registration result based on the at least two second registration results, the method further includes: Obtain a first relationship, which is the relative relationship of respiratory displacements of different parts of the target tissue, wherein the respiratory displacements are displacements generated by the breathing of the target human body; Based on the first relationship, a second relationship is obtained, which is the relative relationship of the breathing displacements of different image blocks in the first image block set; Based on the second relationship, the first image block set is divided into at least two third image block subsets. The difference in respiratory displacement between two image blocks in the same third image block subset is less than or equal to the second threshold, and the difference in respiratory displacement between two image blocks in different third image block subsets is greater than the second threshold. Based on the at least two third image block sets, the second image block set is divided into at least two fourth image block subsets; Using the first model, the corresponding subsets in the at least two third image block subsets and the at least two fourth image block subsets are registered to obtain at least two third registration results; The process of obtaining the target registration result based on the at least two second registration results includes: The target registration result is obtained based on the at least two second registration results and the at least two third registration results.

[0009] In conjunction with any embodiment of this application, obtaining the target registration result based on the at least two second registration results and the at least two third registration results includes: Based on the at least two second registration results, a fourth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the at least two third registration results, a fifth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the fourth registration result and the fifth registration result, the target registration result is obtained.

[0010] In conjunction with any embodiment of this application, obtaining the target registration result based on the fourth registration result and the fifth registration result includes: Based on the fourth registration result, the third three-dimensional image is transformed to obtain the first transformed image; Based on the fifth registration result, the third three-dimensional image is transformed to obtain the second transformed image; If the difference between the first transformed image and the second three-dimensional image is less than the difference between the second transformed image and the second three-dimensional image, the fourth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is greater than the difference between the second transformed image and the second three-dimensional image, the fifth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is equal to the difference between the second transformed image and the second three-dimensional image, the fourth registration result or the fifth registration result shall be used as the target registration result.

[0011] In conjunction with any embodiment of this application, the step of dividing the second three-dimensional image and the third three-dimensional image into at least two image blocks according to the same division method to obtain a first image block set and a second image block set includes: The third three-dimensional image is cropped to obtain a fourth three-dimensional image, wherein the target tissue in the fourth three-dimensional image is the same as the target tissue in the third three-dimensional image, and the size of the fourth three-dimensional image is the same as the size of the second three-dimensional image; Following the same division method, the second three-dimensional image and the fourth three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set.

[0012] In any embodiment of this application, the first three-dimensional image is acquired before the target time, the first three-dimensional image is used to determine whether there is a lesion in the target tissue, the second three-dimensional image is used to determine the target path, the target path is the path from the skin area of ​​the target human body to the lesion, and the target tissue is the tissue in the target human body; The method further includes: Based on the target registration result, the third 3D image is transformed to obtain the third transformed image; Key regions are determined from the third transformed image, and the key regions include at least one of the following: blood vessel region and bone region, wherein the blood vessel region is the region corresponding to the blood vessels of the target human body, and the bone region is the region corresponding to the bones of the target human body; Based on the position of the key region in the third transformed image, a region to be enhanced is determined from the second three-dimensional image, wherein the position of the region to be enhanced in the second three-dimensional image is the same as the position of the key region in the third transformed image; Based on the key region, the key information in the region to be enhanced in the second three-dimensional image is enhanced to obtain an enhanced image. The key information includes at least one of the following: information related to the blood vessels of the target human body, and information related to the bones of the target human body. The target path is obtained based on the enhanced image.

[0013] Secondly, a semantic-based registration device is provided, the semantic-based registration device comprising: The acquisition unit is used to acquire a first three-dimensional image and a second three-dimensional image, both of which include the target tissue. The difference between the volume of the target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image is greater than or equal to a first threshold. The processing unit is configured to remove regions in the first three-dimensional image that are different from the target tissue in the second three-dimensional image based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, thereby obtaining a third three-dimensional image. The processing unit is further configured to use a first model to register the second three-dimensional image and the third three-dimensional image to obtain a target registration result. The first model is trained based on a target loss function, the function value of which is positively correlated with the volume difference. The volume difference indicates the difference in volume of the target tissue in the two three-dimensional images to be registered. The target registration result is used to align the second three-dimensional image and the third three-dimensional image.

[0014] In conjunction with any embodiment of this application, the processing unit is further configured to: Using the same division method, the second three-dimensional image and the third three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set; The corresponding image blocks in the first image block set and the second image block set are registered to obtain a first registration result set, and the first registration result in the first registration result set corresponds one-to-one with the image blocks in the first image block set; Cluster the first registration results in the first registration result set to obtain at least two cluster sets; Based on the at least two cluster sets, the first image patch set is divided into at least two first image patch subsets, and the second image patch set is divided into at least two second image patch subsets. The first registration results corresponding to the image patches in the first image patch subsets belong to the same cluster set, and the first registration results corresponding to the image patches in the second image patch subsets belong to another cluster set. Using the first model, the corresponding subsets in the at least two first image block subsets and the at least two second image block subsets are registered to obtain at least two second registration results; The target registration result is obtained based on the at least two second registration results.

[0015] In any embodiment of this application, the target tissue belongs to the target human body; The acquisition unit is further configured to acquire a first relationship, which is the relative relationship of respiratory displacements of different parts of the target tissue, wherein the respiratory displacement is the displacement generated by the respiration of the target human body. The processing unit is further configured to: Based on the first relationship, a second relationship is obtained, which is the relative relationship of the breathing displacements of different image blocks in the first image block set; Based on the second relationship, the first image block set is divided into at least two third image block subsets. The difference in respiratory displacement between two image blocks in the same third image block subset is less than or equal to the second threshold, and the difference in respiratory displacement between two image blocks in different third image block subsets is greater than the second threshold. Based on the at least two third image block sets, the second image block set is divided into at least two fourth image block subsets; Using the first model, the corresponding subsets in the at least two third image block subsets and the at least two fourth image block subsets are registered to obtain at least two third registration results; The target registration result is obtained based on the at least two second registration results and the at least two third registration results.

[0016] In conjunction with any embodiment of this application, the processing unit is further configured to: Based on the at least two second registration results, a fourth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the at least two third registration results, a fifth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the fourth registration result and the fifth registration result, the target registration result is obtained.

[0017] In conjunction with any embodiment of this application, the processing unit is further configured to: Based on the fourth registration result, the third three-dimensional image is transformed to obtain the first transformed image; Based on the fifth registration result, the third three-dimensional image is transformed to obtain the second transformed image; If the difference between the first transformed image and the second three-dimensional image is less than the difference between the second transformed image and the second three-dimensional image, the fourth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is greater than the difference between the second transformed image and the second three-dimensional image, the fifth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is equal to the difference between the second transformed image and the second three-dimensional image, the fourth registration result or the fifth registration result shall be used as the target registration result.

[0018] In conjunction with any embodiment of this application, the processing unit is further configured to: The third three-dimensional image is cropped to obtain a fourth three-dimensional image, wherein the target tissue in the fourth three-dimensional image is the same as the target tissue in the third three-dimensional image, and the size of the fourth three-dimensional image is the same as the size of the second three-dimensional image; Following the same division method, the second three-dimensional image and the fourth three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set.

[0019] In any embodiment of this application, the first three-dimensional image is acquired before the target time, the first three-dimensional image is used to determine whether there is a lesion in the target tissue, the second three-dimensional image is used to determine the target path, the target path is the path from the skin area of ​​the target human body to the lesion, and the target tissue is the tissue in the target human body; The processing unit is further configured to: Based on the target registration result, the third 3D image is transformed to obtain the third transformed image; Key regions are determined from the third transformed image, and the key regions include at least one of the following: blood vessel region and bone region, wherein the blood vessel region is the region corresponding to the blood vessels of the target human body, and the bone region is the region corresponding to the bones of the target human body; Based on the position of the key region in the third transformed image, a region to be enhanced is determined from the second three-dimensional image, wherein the position of the region to be enhanced in the second three-dimensional image is the same as the position of the key region in the third transformed image; Based on the key region, the key information in the region to be enhanced in the second three-dimensional image is enhanced to obtain an enhanced image. The key information includes at least one of the following: information related to the blood vessels of the target human body, and information related to the bones of the target human body. The target path is obtained based on the enhanced image.

[0020] Thirdly, a surgical robot is provided, including a semantic-based registration device as described in the second aspect. In this third aspect, the surgical robot can perform a semantic-based registration method using the semantic-based registration device, thereby improving the accuracy of the target registration results.

[0021] Fourthly, an electronic device is provided, comprising: a processor and a memory, the memory for storing computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0022] Fifthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0023] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method as described in the first aspect above and any possible implementation thereof.

[0024] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, wherein, when the computer program or instructions are executed on a computer, the computer performs the method described in the first aspect and any possible implementation thereof.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0026] In this embodiment, both the first and second 3D images include the target tissue, and the difference between the volume of the target tissue in the first 3D image and the volume of the target tissue in the second 3D image is greater than or equal to a first threshold, meaning the difference between the target tissue in the first and second 3D images is significant. After acquiring the first and second 3D images, the registration device removes regions in the first and second 3D images that differ from the target tissue in the second 3D image based on the semantics of voxels in the first and second 3D images, thus obtaining a third 3D image. This reduces the difference between the target tissue in the first and second 3D images. Then, the first model is used to register the second and third 3D images to obtain the target registration result, which reduces interference caused by differences in the target tissue and improves the accuracy of the target registration result. Moreover, since the first model is trained based on a target loss function, and the function value of the target loss function is positively correlated with the volume difference, obtaining the target registration result based on the first model can improve the accuracy of the target registration result. In summary, based on Figure 1 The method used to obtain target registration results can improve the accuracy of target registration results. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be described below.

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0029] Figure 1 A flowchart illustrating a semantic-based registration method provided in an embodiment of this application; Figure 2 A flowchart illustrating a training method provided in an embodiment of this application; Figure 3a A schematic diagram of a third three-dimensional image provided in an embodiment of this application; Figure 3b A schematic diagram of a second three-dimensional image provided in an embodiment of this application; Figure 3c A schematic diagram illustrating a target registration result provided in an embodiment of this application; Figure 3d A schematic diagram of a third transformed image provided in an embodiment of this application; Figure 4a A schematic diagram of a third three-dimensional image provided in an embodiment of this application; Figure 4b A schematic diagram of a second three-dimensional image provided in an embodiment of this application; Figure 4c A schematic diagram illustrating a target registration result provided in an embodiment of this application; Figure 4d A schematic diagram of a third transformed image provided in an embodiment of this application; Figure 5a A schematic diagram of yet another third three-dimensional image provided in an embodiment of this application; Figure 5b A schematic diagram of yet another second three-dimensional image provided in an embodiment of this application; Figure 5c A schematic diagram of yet another third-transformed image provided in an embodiment of this application; Figure 6a A schematic diagram of yet another third three-dimensional image provided in an embodiment of this application; Figure 6b A schematic diagram of yet another second three-dimensional image provided in an embodiment of this application; Figure 6c A schematic diagram of yet another third-transformed image provided in an embodiment of this application; Figure 7 A schematic diagram of the structure of a semantic-based registration device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0031] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0032] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. It should be understood that in this application, "at least one" means one or more, "more" means two or more, and "at least two" means two or three or more.

[0033] The execution subject of this application embodiment is a semantic registration device (hereinafter referred to as a registration device), wherein the registration device can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the registration device can be one of the following: a computer, a platform server.

[0034] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a semantic-based registration method provided in an embodiment of this application.

[0035] 101. Obtain the first three-dimensional image and the second three-dimensional image.

[0036] In this embodiment, both the first and second three-dimensional images include the target tissue. The difference between the volume of the target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image is greater than or equal to a first threshold. That is, the target tissue in the first three-dimensional image differs significantly from the target tissue in the second three-dimensional image. In other words, the image content corresponding to the target tissue in the first three-dimensional image is more than the image content corresponding to the target tissue in the second three-dimensional image. For example, the first three-dimensional image is a three-dimensional CT image, the second three-dimensional image is a three-dimensional CBCT image, and the target tissue is the lung. The three-dimensional CT image includes both the left and right lungs, and the three-dimensional CBCT image includes only the left lung; in this case, the three-dimensional CT image includes more image content of the right lung than the three-dimensional CBCT image.

[0037] In some implementations, the first threshold is determined based on the absolute value of the difference between the volume of the target tissue in the first historical 3D image and the volume of the target tissue in the second historical 3D image, wherein the difference between the target tissue in the first historical 3D image and the target tissue in the second historical 3D image is small. For example, the time difference between the acquisition time of the first historical 3D image and the acquisition time of the second historical 3D image is less than a manually set time threshold.

[0038] Optionally, after acquiring at least one pair of first historical 3D images, the absolute value of the difference in volume between the target tissues in the at least one pair of first historical 3D images is determined to obtain at least one volume difference. A pair of first historical 3D images includes one first historical 3D image and one second historical 3D image, and the volume difference is the absolute value of the difference between the volume of the target tissue in the first historical 3D image and the volume of the target tissue in the second historical 3D image. The average value of the at least one volume difference is then determined as a first threshold.

[0039] In this embodiment, both the first three-dimensional image and the second three-dimensional image are medical images, and both can be one of the following: ultrasound image, computed tomography (CT) image, or radiograph. Optionally, the first three-dimensional image is a three-dimensional CT image, and the second three-dimensional image is a three-dimensional cone beam computed tomography (CBCT) image.

[0040] The target tissue can be any tissue within the target body, such as a blood vessel, an organ, or cartilage. Optionally, the target tissue could be the lung.

[0041] 102. Based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, remove regions in the first three-dimensional image that are different from the target tissue in the second three-dimensional image to obtain the third three-dimensional image.

[0042] In this embodiment, a voxel is a pixel in a 3D image. The semantics of the voxels in the 3D image can be obtained based on semantic segmentation technology. Optionally, the registration device processes the 3D image using a semantic segmentation model to determine the semantics of the voxels in the 3D image. For example, if the target tissue is the lung, inputting the 3D image into the semantic segmentation model can yield the semantics of the voxels in the 3D image as: left lung, right lung, or background, where background refers to something other than the lung. Optionally, the semantic segmentation model is a U-Net.

[0043] Optionally, before inputting the first and second 3D images into the semantic segmentation model, the first and second 3D images are cropped to ensure that their sizes are both preset, where the preset size is the size of the image processed by the semantic segmentation model. Then, the semantic segmentation model is used to perform semantic segmentation on the first and second 3D images respectively, which can improve the accuracy of the semantic segmentation results.

[0044] Because the differences in the content of the two 3D images to be registered can interfere with the configuration of the two images, resulting in low accuracy of the registration result, the registration device first removes the regions in the first 3D image that are different from the target tissues in the second 3D image by executing step 102 before registering the first 3D image and the second 3D image, thus obtaining the third 3D image. This can reduce the difference between the target tissues in the first 3D image and the target tissues in the second 3D image.

[0045] In one possible implementation, the target tissue is the lung, with a first 3D image including both the left and right lungs, and a second 3D image including the left lung. The registration device obtains a third 3D image by removing the region corresponding to the right lung from the first 3D image.

[0046] The second 3D image includes the left lung. The registration device obtains the third 3D image by removing the region corresponding to the right lung from the first 3D image.

[0047] 103. Using the first model, register the second and third 3D images to obtain the target registration result.

[0048] In this embodiment, the first model is trained based on a target loss function. The value of the target loss function is positively correlated with the volume difference, which indicates the difference in volume of the target tissue in the two 3D images to be registered. In other words, during training, the first model adjusts its parameters with the optimization direction of reducing the volume difference of the target tissue in the two 3D images to be registered. Because the target tissue in the two 3D images has the same volume when aligned, training the first model based on the target loss function can improve the accuracy of the registration results obtained by the first model.

[0049] The target registration results are used to align the second and third 3D images. Optionally, the target registration results are used to align target tissues in the second and third 3D images.

[0050] Optionally, the target tissue is the lung. The first 3D image includes both the left and right lungs, and the second 3D image also includes both the left and right lungs. In the first 3D image, the left lung is complete, while in the second 3D image, the left lung is incomplete. In both the first and second 3D images, the right lung is complete. In this case, the region in the first 3D image that differs from the target tissue in the second 3D image includes the left lung. The registration device obtains the third 3D image by retaining the right lung in the first 3D image. By retaining the right lung in the second 3D image, a removed second 3D image is obtained. Using the first model, the removed second and third 3D images are registered to obtain the target registration result. This target registration result is then used to align the removed second and third 3D images.

[0051] In one possible implementation method Figure 2 This is a flowchart illustrating a training method provided in an embodiment of this application. The first model can be... Figure 2 The training method shown is used to obtain the image. Specifically, after the process begins, a first training 3D image and a second training 3D image are acquired, both of which include a lung, where the lung is the target tissue. Optionally, the difference between the lung volume in the first training 3D image and the lung volume in the second training 3D image is less than or equal to a first threshold. Optionally, the size of the first training 3D image is the same as the size of the second training 3D image.

[0052] Semantic segmentation is performed on the first and second training 3D images respectively, yielding a first semantic segmentation result for the first training 3D image and a second semantic segmentation result for the second training 3D image. Based on the second semantic segmentation result, the left and right lungs in the second training 3D image, as well as their volumes, can be determined. Then, based on the volumes of the left and right lungs, it can be determined whether the left or right lung is complete; specifically, the volume of the left lung determines whether the left lung is complete, and the volume of the right lung determines whether the right lung is complete. Then, the regions corresponding to incomplete lungs are removed, and the regions corresponding to complete lungs are retained, resulting in the third training 3D image. Simultaneously, the regions corresponding to incomplete lungs in the first training 3D image are removed, resulting in the fourth training 3D image. For example, based on the volume of the left lung in the second training 3D image, it is determined that the left lung is incomplete; in this case, the regions corresponding to the left lung in the first training 3D image are removed to obtain the fourth training 3D image. Then, registration is performed based on the second model. Specifically, the third and fourth training 3D images are input into the second model so that the second model can register the third and fourth training 3D images, resulting in the training registration results for the third and fourth training 3D images. Based on the training registration results, the third training 3D image is transformed to obtain the transformed third training 3D image. The absolute value of the difference between the lung volume in the transformed fourth training 3D image and the lung volume in the transformed third training 3D image is determined. If this absolute value is greater than a third threshold, the value of the target loss function is determined. Specifically, based on the lung volume in the transformed third training 3D image, the difference in lung volume in the fourth training 3D image, and the target loss function, the value of the target loss function is obtained, where the function value is positively correlated with the absolute value. Optionally, the target loss function can be a function for calculating the mean-square error (MSE) or a function for calculating the mean absolute error (MAE). Based on the function value of the target loss function, update the parameters of the second model until the condition for stopping the update is met. Then, stop updating the parameters of the second model and use the second model as the first model. The condition for stopping the update includes at least one of the following: the absolute value of the difference between the lung volume in the transformed third training 3D image and the lung volume in the fourth training 3D image is less than or equal to a third threshold, or the number of times the parameters of the second model are updated reaches a preset number.

[0053] exist Figure 1In the semantic-based registration method, both the first and second 3D images include the target tissue, and the difference between the volume of the target tissue in the first 3D image and the volume of the target tissue in the second 3D image is greater than or equal to a first threshold, meaning there is a significant difference between the target tissue in the first and second 3D images. After acquiring the first and second 3D images, the registration device removes regions in the first and second 3D images that differ from the target tissue in the second 3D image based on the semantics of voxels in the first and second 3D images, thus obtaining a third 3D image. This reduces the difference between the target tissue in the first and second 3D images. Then, the first model is used to register the second and third 3D images to obtain the target registration result, which reduces interference caused by differences in the target tissue and improves the accuracy of the target registration result. Moreover, since the first model is trained based on a target loss function, and the function value of the target loss function is positively correlated with the volume difference, obtaining the target registration result based on the first model can improve the accuracy of the target registration result. In summary, based on... Figure 1 The method used to obtain target registration results can improve the accuracy of target registration results.

[0054] As an optional implementation, before registering the second and third 3D images using the first model to obtain the target registration result, the registration device further performs the following steps: Dividing the second and third 3D images into at least two image blocks according to the same partitioning method, obtaining a first image block set and a second image block set. Registering the corresponding image blocks in the first and second image block sets to obtain a first registration result set, wherein the first registration result in the first registration result set corresponds one-to-one with the image blocks in the first image block set. Clustering the first registration results in the first registration result set to obtain at least two cluster sets. Based on the at least two cluster sets, dividing the first image block set into at least two first image block subsets, and dividing the second image block set into at least two second image block subsets, wherein the first registration results corresponding to the image blocks in the first image block subsets belong to the same cluster set, and the first registration results corresponding to the image blocks in the second image block subsets belong to another cluster set.

[0055] After obtaining at least two first image patch subsets and at least two second image patch subsets, the registration device performs the following steps during the execution of step "registering the second 3D image and the third 3D image using the first model to obtain the target registration result": using the first model, registering corresponding subsets from the at least two first image patch subsets and at least two second image patch subsets to obtain at least two second registration results. Based on the at least two second registration results, the target registration result is obtained.

[0056] Because tissues within the human body move due to respiration, the same tissue can appear in different 3D images. Therefore, the difference in the position of the target tissue in the first 3D image and the second 3D image is primarily caused by respiration. Furthermore, because different parts of the target tissue are affected by respiration differently, the amplitude of the respiration-induced movement varies among these parts. Therefore, in this embodiment, the registration device first divides the second and third 3D images into at least two image blocks using the same partitioning method, resulting in a first image block set and a second image block set. In the subsequent registration process, by registering the corresponding image blocks from the first and second image block sets, the impact of the differences in the amplitude of movement of different parts of the target tissue on the registration result can be reduced, thereby improving the accuracy of the registration result.

[0057] To reduce the impact of differences in motion amplitude between different parts on the registration results, the registration device first registers corresponding image blocks in the first image block set and the second image block set, obtaining a first registration result set. Since the first registration result set includes the registration results corresponding to image blocks in the first image block set, and the registration results corresponding to image blocks are related to the motion amplitude of the part corresponding to that image block, the registration device clusters the first registration results in the first registration result set after obtaining it, obtaining at least two cluster sets. Optionally, the difference between two first registration results in the same cluster set is less than or equal to a fourth threshold, and the difference between two first registration results in different cluster sets is greater than the fourth threshold. For example, if the first registration result is a deformation field, the difference between two first registration results can be the MSE of the two deformation fields or the MAE of the two deformation fields. For another example, first registration result p1 is the registration result of image block i1 and image block i2, and first registration result p2 is the registration result of image block i3 and image block i4. Image block i1 is transformed based on the first registration result p1 to obtain image block i5, and image block i3 is transformed based on the first registration result p2 to obtain image block i6. The mutual information value h1 between image block i5 and image block i2 is determined, and the mutual information value h2 between image block i6 and image block i4 is determined. The difference between the first registration result p1 and the first registration result p2 is obtained by comparing the mutual information values ​​h1 and h2.

[0058] Because there is a correspondence between the first registration result in at least two cluster sets and the image blocks in the first image block set and the second image block set, the registration device, after obtaining at least two cluster sets, classifies the image blocks in the first image block set and the image blocks in the second image block set based on the at least two cluster sets, thus obtaining a first image block subset and a second image block subset. For example, the first image block set includes image block A1, image block A2, image block A3, and image block A4, and the second image block set includes image block A5, image block A6, image block A7, and image block A8. The registration result between image block A1 and image block A5 is the first registration result B1, the registration result between image block A2 and image block A6 is the first registration result B2, the registration result between image block A3 and image block A7 is the first registration result B3, and the registration result between image block A4 and image block A8 is the first registration result B4. If the first registration results B1 and B2 belong to set J1 of at least two cluster sets, and the first registration results B3 and B4 belong to set J2 of at least two cluster sets, then image patches A1 and A2 belong to the first sub-image patch set J3, image patches A3 and A4 belong to the first sub-image patch set J4, image patches A5 and A6 belong to the second sub-image patch set J5, and image patches A7 and A8 belong to the second sub-image patch set J6. Specifically, the first image patch set J3 corresponds to the second image patch set J5, and the first image patch set J4 corresponds to the second image patch set J6.

[0059] After obtaining at least two first image block subsets and at least two second image block subsets, the registration device uses a first model to register corresponding subsets within the at least two first image block subsets and at least two second image block subsets, obtaining at least two second registration results. Then, based on these at least two second registration results, a target registration result is obtained. This reduces the interference of differences in motion amplitude between different parts on the registration, thereby improving the accuracy of the at least two second registration results, and consequently improving the accuracy of the target registration result.

[0060] Optionally, the first model obtains the target registration result based on at least two second registration results.

[0061] Optionally, the registration device uses the first model based on B-spline curves and at least two third registration results to obtain a global registration result of the second and third three-dimensional images, which serves as the target registration result.

[0062] As an optional implementation, the target tissue belongs to the target human body. Before obtaining the target registration result based on at least two second registration results, the registration device further performs the following steps: obtaining a first relationship, wherein the first relationship is the relative relationship of respiratory displacements of different parts in the target tissue, and the respiratory displacement is the displacement generated by the breathing of the target human body. Based on the first relationship, obtaining a second relationship, wherein the second relationship is the relative relationship of respiratory displacements of different image blocks in a first image block set. Based on the second relationship, dividing the first image block set into at least two third image block subsets, wherein the difference in respiratory displacements between two image blocks in the same third image block subset is less than or equal to a second threshold, and the difference in respiratory displacements between two image blocks in different third image block subsets is greater than the second threshold. Based on the at least two third image block sets, dividing the second image block set into at least two fourth image block subsets. Using a first model, registering the corresponding subsets in the at least two third image block subsets and the at least two fourth image block subsets to obtain at least two third registration results.

[0063] In some implementations, the second threshold is determined based on the absolute value of the difference between the respiratory displacement of the target tissue in the third historical 3D image and the respiratory displacement of the target tissue in the fourth historical 3D image, where the target tissue in the third historical 3D image and the target tissue in the fourth historical 3D image correspond to the same region within the target tissue. For example, if the target tissue is the lung, which can be divided into upper left, lower left, upper right, and lower right regions, the target tissue in both the third and fourth historical 3D images corresponds to the upper left region; that is, both the third and fourth historical 3D images include the upper left region of the lung.

[0064] Optionally, after acquiring at least one pair of second historical image blocks, the absolute value of the difference in respiratory displacement of the target tissue in the at least one pair of second historical image blocks is determined to obtain at least one respiratory displacement difference. Here, a pair of second historical image blocks includes one third historical 3D image and one fourth historical 3D image, and the respiratory displacement difference is the absolute value of the difference between the respiratory displacement of the target tissue in the third historical 3D image and the respiratory displacement of the target tissue in the fourth historical 3D image. The average value of the at least one respiratory displacement difference is then determined as a second threshold.

[0065] After obtaining at least two third registration results, the following steps are performed during the execution of the step "obtain the target registration result based on at least two second registration results": obtain the target registration result based on at least two second registration results and at least two third registration results.

[0066] In this embodiment, respiratory displacement is the displacement generated by respiration, and therefore, respiratory displacement can reflect the amplitude of movement caused by respiration. As mentioned above, the amplitude of movement caused by respiration varies in different parts of the target tissue. Therefore, after obtaining the second relationship based on the first relationship, the registration device classifies the image blocks in the first image block set based on the second relationship to obtain the third image block subset.

[0067] For example, the target tissue includes parts c1, c2, c3, and c4. A first relationship states that the respiratory displacement of part c2 is 1 mm smaller than that of part c1, the respiratory displacement of part c3 is 1.5 mm smaller than that of part c1, and the respiratory displacement of part c4 is 2 mm smaller than that of part c1. A first set of image blocks includes image block A1, image block A2, image block A3, and image block A4, where image block A1 corresponds to part c1, image block A2 corresponds to part c2, image block A3 corresponds to part c3, and image block A4 corresponds to part c4. Therefore, a second relationship derived from the first relationship states that the respiratory displacement of image block A2 is 1 mm smaller than that of image block A1, the respiratory displacement of image block A3 is 1.5 mm smaller than that of image block A1, and the respiratory displacement of image block A4 is 2 mm smaller than that of image block A1.

[0068] If the classification of image blocks in the first image block set based on the second relation is as follows: image block A1 and image blocks whose breathing displacement differs from that of image block A1 by less than or equal to 0.5 mm are classified into one category; image blocks whose breathing displacement differs from that of image block A1 by more than 0.5 mm are classified into another category. Then, based on the second relation, the first image block set can be divided into a third image block subset J7 and a third image block subset J8. The third image block subset J7 includes image blocks A1 and A2, and the third image block subset J8 includes image blocks A3 and A4.

[0069] Optionally, the first relationship is pre-calibrated. For example, the first relationship can be obtained by statistically analyzing the respiratory displacements of different parts of the target tissue.

[0070] After obtaining at least two third image subsets, the registration device divides the second image block set into at least two fourth image block subsets based on these at least two third image subsets. That is, it divides the second image block set into at least two fourth image block subsets based on the classification method used for the first image set. Then, using a first model, corresponding subsets within the at least two third and at least two fourth image block subsets are registered to obtain at least two third registration results. These at least two third registration results are registration results obtained based on a first relation. Finally, based on the at least two second and at least two third registration results, the target registration result is obtained. This reduces the interference of differences in motion amplitude between different parts on the registration, thereby improving the accuracy of the target registration result.

[0071] Optionally, the registration device obtains the target registration result based on at least two third registration results, which can reduce the interference of the difference in motion amplitude of different parts on the registration, thereby improving the accuracy of the at least two third registration results, and thus improving the accuracy of the target registration result.

[0072] As an optional implementation, obtaining a target registration result based on at least two second registration results and at least two third registration results includes: obtaining a fourth registration result based on at least two second registration results, wherein the fourth registration result is used to align the second 3D image and the third 3D image; obtaining a fifth registration result based on at least two third registration results, wherein the fifth registration result is used to align the second 3D image and the third 3D image; and obtaining the target registration result based on the fourth and fifth registration results.

[0073] In this embodiment, the registration device first obtains a global registration result (i.e., a fourth registration result) of the second and third three-dimensional images based on at least two second registration results, and obtains a global registration result (i.e., a fifth registration result) of the second and third three-dimensional images based on at least two third registration results. Then, based on the fourth and fifth registration results, a target registration result is obtained, which can improve the accuracy of the target registration result.

[0074] Optionally, the registration device determines the average value of the fourth and fifth registration results to obtain the target registration result.

[0075] As an optional implementation, obtaining a target registration result based on the fourth and fifth registration results includes the following steps: Transforming the third 3D image based on the fourth registration result to obtain a first transformed image. Transforming the third 3D image based on the fifth registration result to obtain a second transformed image. If the difference between the first transformed image and the second 3D image is less than the difference between the second transformed image and the second 3D image, the fourth registration result is used as the target registration result. If the difference between the first transformed image and the second 3D image is greater than the difference between the second transformed image and the second 3D image, the fifth registration result is used as the target registration result. If the difference between the first transformed image and the second 3D image is equal to the difference between the second transformed image and the second 3D image, either the fourth or fifth registration result is used as the target registration result.

[0076] In this embodiment, the difference between the first transformed image and the second three-dimensional image represents the error of the fourth registration result, and the difference between the second transformed image and the second three-dimensional image represents the error of the fifth registration result. Through this embodiment, the registration device can use the registration result with the smaller error between the fourth and fifth registration results as the target registration result, thereby improving the accuracy of the target registration result.

[0077] As an optional implementation, the second and third three-dimensional images are divided into at least two image blocks according to the same partitioning method to obtain a first image block set and a second image block set. This includes the following steps: cropping the third three-dimensional image to obtain a fourth three-dimensional image, wherein the target tissue in the fourth three-dimensional image is the same as the target tissue in the third three-dimensional image, and the size of the fourth three-dimensional image is the same as the size of the second three-dimensional image. The second and fourth three-dimensional images are divided into at least two image blocks according to the same partitioning method to obtain a first image block set and a second image block set.

[0078] In this embodiment, after the registration device obtains the fourth three-dimensional image by cropping the third three-dimensional image, it divides the second three-dimensional image and the fourth three-dimensional image into a first image block set and a second image block set, thereby improving the matching degree of the first image block set and the second image block set, which is beneficial for subsequent registration of the first image block set and the second image block set.

[0079] As an optional implementation, the first three-dimensional image is acquired before the target time, and is used to determine whether a lesion exists within the target tissue. The second three-dimensional image is used to determine the target path, where the target path is the path from the skin region of the target human body to the lesion, and the target tissue is the tissue within the target human body.

[0080] In some scenarios, a CT scan of the human body yields a first 3D image, which can then be used to determine whether lesions exist in target tissues. If a lesion is confirmed, a CBCT scan is performed to obtain a second 3D image. The target path, defined as the path from the skin region of the target body to the lesion, is then determined based on this second 3D image. In this scenario, because CT images have high quality (e.g., high resolution), determining the presence of lesions in target tissues based on CT images improves accuracy. Furthermore, CBCT scans involve less radiation, thus minimizing radiation exposure during target path determination. However, since the quality of 3D CBCT images is lower than that of 3D CT images, registering the 3D CT image (i.e., the first 3D image) and the 3D CBCT image (i.e., the second 3D image) helps in determining the target path based on the 3D CT image, compensating for the lower quality of the 3D CBCT image and improving the accuracy of the target path determination.

[0081] In this embodiment, the registration device further performs the following steps: Based on the target registration result, transform the third three-dimensional image to obtain a third transformed image. Determine key regions from the third transformed image, the key regions including at least one of the following: a blood vessel region and a bone region, wherein the blood vessel region corresponds to the blood vessels of the target human body, and the bone region corresponds to the bones of the target human body. Based on the position of the key regions in the third transformed image, determine the region to be enhanced from the second three-dimensional image, wherein the position of the region to be enhanced in the second three-dimensional image is the same as the position of the key regions in the third transformed image. Based on the key regions, enhance the key information in the region to be enhanced in the second three-dimensional image to obtain an enhanced image, wherein the key information includes at least one of the following: information related to the blood vessels of the target human body, and information related to the bones of the target human body. Based on the enhanced image, obtain the target path.

[0082] The registration device transforms the third 3D image based on the second registration result to align the third 3D image with the second 3D image, obtaining a third transformed image. Then, based on the key regions in the third transformed image, it enhances the key information of the regions to be enhanced in the second 3D image, obtaining an enhanced image. Finally, based on the enhanced image, the target path is obtained, reducing the probability of the target path intersecting with the key regions.

[0083] For example, Figure 3a This is a schematic diagram of a third three-dimensional image provided in an embodiment of this application. Figure 3b This is a schematic diagram of a second three-dimensional image provided in an embodiment of this application. Figure 3c This is a schematic diagram illustrating a target registration result provided in an embodiment of this application. Figure 3d This is a schematic diagram of a third transformed image provided in an embodiment of this application. Wherein, Figure 3c The target registration results shown are Figure 3a The third three-dimensional image shown is... Figure 3b The registration result of the second 3D image is shown. Specifically, the target registration result is the deformation field. Figure 3d To base the target registration results on Figure 3a The image obtained by transforming the third 3D image shown is the third transformed image.

[0084] For example, Figure 4a This is a schematic diagram of a third three-dimensional image provided in an embodiment of this application. Figure 4b This is a schematic diagram of a second three-dimensional image provided in an embodiment of this application. Figure 4c This is a schematic diagram illustrating a target registration result provided in an embodiment of this application. Figure 4d This is a schematic diagram of a third transformed image provided in an embodiment of this application. Wherein, Figure 4c The target registration results shown are Figure 4a The third three-dimensional image shown is... Figure 4b The registration result of the second 3D image is shown. Specifically, the target registration result is the deformation field. Figure 4d To base the target registration results on Figure 4a The image obtained by transforming the third 3D image shown is the third transformed image.

[0085] For example, Figure 5a This is a schematic diagram of yet another third-dimensional image provided in an embodiment of this application. Figure 5b This is a schematic diagram of yet another second three-dimensional image provided in an embodiment of this application. Figure 5c This is a schematic diagram of yet another third-transformed image provided in an embodiment of this application. Wherein, Figure 5a The target organization and Figure 5b The target tissue in the study was the lungs. Figure 5a For 3D CT images, Figure 5b This is a 3D CBCT image. Figure 5c To base the target registration results on Figure 5a The image obtained by transforming the third 3D image shown is the third transformed image.

[0086] For example, Figure 6a This is a schematic diagram of yet another third-dimensional image provided in an embodiment of this application. Figure 6b This is a schematic diagram of yet another second three-dimensional image provided in an embodiment of this application. Figure 6c This is a schematic diagram of yet another third-transformed image provided in an embodiment of this application. Wherein, Figure 6a The target organization and Figure 6bThe target tissue in the study was the lungs. Figure 6a For 3D CT images, Figure 6b This is a 3D CBCT image. Figure 6c To base the target registration results on Figure 6a The image obtained by transforming the third 3D image shown is the third transformed image.

[0087] Optionally, the registration device overlays the key region with the region to be enhanced in the second 3D image to obtain the enhanced image.

[0088] It should be understood that if the target registration result is the fourth registration result, then the third transformed image is the first transformed image. If the target registration result is the fifth registration result, then the third transformed image is the second transformed image.

[0089] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0090] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0091] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0092] Please see Figure 7 , Figure 7 This is a schematic diagram of a semantic-based registration device provided in an embodiment of this application. The semantic-based registration device 1 includes: an acquisition unit 11 and a processing unit 12, wherein: The acquisition unit 11 is used to acquire a first three-dimensional image and a second three-dimensional image, both of which include the target tissue. The difference between the volume of the target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image is greater than or equal to a first threshold. Processing unit 12 is used to remove regions in the first three-dimensional image that are different from the target tissue in the second three-dimensional image based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, to obtain a third three-dimensional image; The processing unit 12 is further configured to use a first model to register the second three-dimensional image and the third three-dimensional image to obtain a target registration result. The first model is trained based on a target loss function, the function value of which is positively correlated with the volume difference. The volume difference indicates the difference in volume of the target tissue in the two three-dimensional images to be registered. The target registration result is used to align the second three-dimensional image and the third three-dimensional image.

[0093] In conjunction with any embodiment of this application, the processing unit 12 is further configured to: Using the same division method, the second three-dimensional image and the third three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set; The corresponding image blocks in the first image block set and the second image block set are registered to obtain a first registration result set, and the first registration result in the first registration result set corresponds one-to-one with the image blocks in the first image block set; Cluster the first registration results in the first registration result set to obtain at least two cluster sets; Based on the at least two cluster sets, the first image patch set is divided into at least two first image patch subsets, and the second image patch set is divided into at least two second image patch subsets. The first registration results corresponding to the image patches in the first image patch subsets belong to the same cluster set, and the first registration results corresponding to the image patches in the second image patch subsets belong to another cluster set. Using the first model, the corresponding subsets in the at least two first image block subsets and the at least two second image block subsets are registered to obtain at least two second registration results; The target registration result is obtained based on the at least two second registration results.

[0094] In any embodiment of this application, the target tissue belongs to the target human body; The acquisition unit 11 is further configured to acquire a first relationship, which is the relative relationship of respiratory displacements of different parts in the target tissue, wherein the respiratory displacement is the displacement generated by the breathing of the target human body. The processing unit 12 is further configured to: Based on the first relationship, a second relationship is obtained, which is the relative relationship of the breathing displacements of different image blocks in the first image block set; Based on the second relationship, the first image block set is divided into at least two third image block subsets. The difference in respiratory displacement between two image blocks in the same third image block subset is less than or equal to the second threshold, and the difference in respiratory displacement between two image blocks in different third image block subsets is greater than the second threshold. Based on the at least two third image block sets, the second image block set is divided into at least two fourth image block subsets; Using the first model, the corresponding subsets in the at least two third image block subsets and the at least two fourth image block subsets are registered to obtain at least two third registration results; The target registration result is obtained based on the at least two second registration results and the at least two third registration results.

[0095] In conjunction with any embodiment of this application, the processing unit 12 is further configured to: Based on the at least two second registration results, a fourth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the at least two third registration results, a fifth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the fourth registration result and the fifth registration result, the target registration result is obtained.

[0096] In conjunction with any embodiment of this application, the processing unit 12 is further configured to: Based on the fourth registration result, the third three-dimensional image is transformed to obtain the first transformed image; Based on the fifth registration result, the third three-dimensional image is transformed to obtain the second transformed image; If the difference between the first transformed image and the second three-dimensional image is less than the difference between the second transformed image and the second three-dimensional image, the fourth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is greater than the difference between the second transformed image and the second three-dimensional image, the fifth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is equal to the difference between the second transformed image and the second three-dimensional image, the fourth registration result or the fifth registration result shall be used as the target registration result.

[0097] In conjunction with any embodiment of this application, the processing unit 12 is further configured to: The third three-dimensional image is cropped to obtain a fourth three-dimensional image, wherein the target tissue in the fourth three-dimensional image is the same as the target tissue in the third three-dimensional image, and the size of the fourth three-dimensional image is the same as the size of the second three-dimensional image; Following the same division method, the second three-dimensional image and the fourth three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set.

[0098] In any embodiment of this application, the first three-dimensional image is acquired before the target time, the first three-dimensional image is used to determine whether there is a lesion in the target tissue, the second three-dimensional image is used to determine the target path, the target path is the path from the skin area of ​​the target human body to the lesion, and the target tissue is the tissue in the target human body; The processing unit 12 is further configured to: Based on the target registration result, the third 3D image is transformed to obtain the third transformed image; Key regions are determined from the third transformed image, and the key regions include at least one of the following: blood vessel region and bone region, wherein the blood vessel region is the region corresponding to the blood vessels of the target human body, and the bone region is the region corresponding to the bones of the target human body; Based on the position of the key region in the third transformed image, a region to be enhanced is determined from the second three-dimensional image, wherein the position of the region to be enhanced in the second three-dimensional image is the same as the position of the key region in the third transformed image; Based on the key region, the key information in the region to be enhanced in the second three-dimensional image is enhanced to obtain an enhanced image. The key information includes at least one of the following: information related to the blood vessels of the target human body, and information related to the bones of the target human body. The target path is obtained based on the enhanced image.

[0099] In this embodiment, both the first and second 3D images include the target tissue, and the difference between the volume of the target tissue in the first 3D image and the volume of the target tissue in the second 3D image is greater than or equal to a first threshold, meaning the difference between the target tissue in the first and second 3D images is significant. After acquiring the first and second 3D images, the registration device removes regions in the first and second 3D images that differ from the target tissue in the second 3D image based on the semantics of voxels in the first and second 3D images, thus obtaining a third 3D image. This reduces the difference between the target tissue in the first and second 3D images. Then, the first model is used to register the second and third 3D images to obtain the target registration result, which reduces interference caused by differences in the target tissue and improves the accuracy of the target registration result. Moreover, since the first model is trained based on a target loss function, and the function value of the target loss function is positively correlated with the volume difference, obtaining the target registration result based on the first model can improve the accuracy of the target registration result. In summary, based on Figure 1 The method used to obtain target registration results can improve the accuracy of target registration results.

[0100] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0101] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0102] The processor 21 can be one or more graphics processing units (GPUs). If the processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, the processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0103] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0104] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0105] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.

[0106] Understandable, Figure 8 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0107] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0112] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0113] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A semantic-based registration method, characterized in that, The method includes: Acquire a first three-dimensional image and a second three-dimensional image, both of which include target tissue, the target tissue being a target human body, and the difference between the volume of the target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image is greater than or equal to a first threshold. Based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, regions in the first three-dimensional image that are different from the target tissue in the second three-dimensional image are removed to obtain a third three-dimensional image. Using a first model, the second three-dimensional image and the third three-dimensional image are registered to obtain a target registration result. The first model is trained based on a target loss function, the function value of which is positively correlated with the volume difference. The volume difference indicates the difference in volume of the target tissue in the two three-dimensional images to be registered. The target registration result is used to align the second three-dimensional image and the third three-dimensional image. Before registering the second 3D image and the third 3D image to obtain the target registration result, the method further includes: Following the same partitioning method, the second 3D image and the third 3D image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set; the corresponding image blocks in the first image block set and the second image block set are registered to obtain a first registration result set, wherein the first registration result in the first registration result set corresponds one-to-one with the image blocks in the first image block set; the first registration results in the first registration result set are clustered to obtain at least two cluster sets; based on the at least two cluster sets, the first image block set is divided into at least two first image block subsets, and the second image block set is divided into at least two second image block subsets, wherein the first registration result corresponding to the image block in the first image block subset belongs to the same cluster set, and the first registration result corresponding to the image block in the second image block subset belongs to another cluster set; The process of registering the second 3D image and the third 3D image using the first model to obtain the target registration result includes: Using the first model, the corresponding subsets in the at least two first image patch subsets and the at least two second image patch subsets are registered to obtain at least two second registration results; based on the at least two second registration results, the target registration result is obtained. Before obtaining the target registration result based on the at least two second registration results, the method further includes: A first relationship is obtained, which is the relative relationship of respiratory displacements of different parts of the target tissue, wherein the respiratory displacement is the displacement generated by the breathing of the target human body; based on the first relationship, a second relationship is obtained, which is the relative relationship of respiratory displacements of different image blocks in the first image block set; based on the second relationship, the first image block set is divided into at least two third image block subsets, wherein the difference in respiratory displacements between two image blocks in the same third image block subset is less than or equal to a second threshold, and the difference in respiratory displacements between two image blocks in different third image block subsets is greater than the second threshold; based on the at least two third image block subsets, the second image block set is divided into at least two fourth image block subsets; using the first model, corresponding subsets in the at least two third image block subsets and the at least two fourth image block subsets are registered to obtain at least two third registration results; The process of obtaining the target registration result based on the at least two second registration results includes: The target registration result is obtained based on the at least two second registration results and the at least two third registration results.

2. The method according to claim 1, characterized in that, The process of obtaining the target registration result based on the at least two second registration results and the at least two third registration results includes: Based on the at least two second registration results, a fourth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the at least two third registration results, a fifth registration result is obtained, which is used to align the second three-dimensional image and the third three-dimensional image. Based on the fourth registration result and the fifth registration result, the target registration result is obtained.

3. The method according to claim 2, characterized in that, The process of obtaining the target registration result based on the fourth registration result and the fifth registration result includes: Based on the fourth registration result, the third three-dimensional image is transformed to obtain the first transformed image; Based on the fifth registration result, the third three-dimensional image is transformed to obtain the second transformed image; If the difference between the first transformed image and the second three-dimensional image is less than the difference between the second transformed image and the second three-dimensional image, the fourth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is greater than the difference between the second transformed image and the second three-dimensional image, the fifth registration result shall be used as the target registration result. If the difference between the first transformed image and the second three-dimensional image is equal to the difference between the second transformed image and the second three-dimensional image, the fourth registration result or the fifth registration result shall be used as the target registration result.

4. The method according to any one of claims 1 to 3, characterized in that, The second three-dimensional image and the third three-dimensional image are divided into at least two image blocks according to the same division method to obtain a first image block set and a second image block set, including: The third three-dimensional image is cropped to obtain a fourth three-dimensional image, wherein the target tissue in the fourth three-dimensional image is the same as the target tissue in the third three-dimensional image, and the size of the fourth three-dimensional image is the same as the size of the second three-dimensional image; Following the same division method, the second three-dimensional image and the fourth three-dimensional image are respectively divided into at least two image blocks to obtain a first image block set and a second image block set.

5. The method according to any one of claims 1 to 3, characterized in that, The first three-dimensional image was acquired before the target time and is used to determine whether there is a lesion in the target tissue. The second three-dimensional image is used to determine the target path, which is the path from the skin area of ​​the target human body to the lesion. The target tissue is the tissue within the target human body. The method further includes: Based on the target registration result, the third 3D image is transformed to obtain the third transformed image; Key regions are determined from the third transformed image, and the key regions include at least one of the following: blood vessel region and bone region, wherein the blood vessel region is the region corresponding to the blood vessels of the target human body, and the bone region is the region corresponding to the bones of the target human body; Based on the position of the key region in the third transformed image, a region to be enhanced is determined from the second three-dimensional image, wherein the position of the region to be enhanced in the second three-dimensional image is the same as the position of the key region in the third transformed image; Based on the key region, the key information in the region to be enhanced in the second three-dimensional image is enhanced to obtain an enhanced image. The key information includes at least one of the following: information related to the blood vessels of the target human body, and information related to the bones of the target human body. The target path is obtained based on the enhanced image.

6. A semantic-based registration apparatus for performing the semantic-based registration method as described in claim 1, characterized in that, The semantic-based registration device includes: The acquisition unit is used to acquire a first three-dimensional image and a second three-dimensional image, both of which include target tissue, the target tissue being a target human body, and the difference between the volume of the target tissue in the first three-dimensional image and the volume of the target tissue in the second three-dimensional image being greater than or equal to a first threshold. The processing unit is configured to remove regions in the first three-dimensional image that are different from the target tissue in the second three-dimensional image based on the semantics of voxels in the first three-dimensional image and the semantics of voxels in the second three-dimensional image, thereby obtaining a third three-dimensional image. The processing unit is further configured to use a first model to register the second three-dimensional image and the third three-dimensional image to obtain a target registration result. The first model is trained based on a target loss function, the function value of which is positively correlated with the volume difference. The volume difference indicates the difference in volume of the target tissue in the two three-dimensional images to be registered. The target registration result is used to align the second three-dimensional image and the third three-dimensional image.

7. A surgical robot, characterized in that, Includes the semantic-based registration device as described in claim 6.

8. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 5.

Images