Method and device for planning a path based on multi-modal data, surgical robot, electronic device

Abstract

The application discloses a method and device for planning a path based on multi-modal data, a surgical robot and electronic equipment. The method comprises the following steps: acquiring a first CT image, the first CT image comprising a target organ, the first CT image not comprising contrast information, the contrast information being generated in the presence of a contrast agent; acquiring a first model; inputting the first CT image into the first model to obtain third information related to the target organ, the third information comprising the contrast information; and performing three-dimensional reconstruction on the target organ based on the third information to obtain a first reconstruction result. The method provided in the application can improve the accuracy of the three-dimensional reconstruction result.

Description

Technical Field

[0001] This application relates to the field of medical imaging technology, and in particular to a method and apparatus for planning paths based on multimodal data, a surgical robot, and electronic equipment. Background Technology

[0002] Three-dimensional organ reconstruction is a technique that uses medical images to reconstruct the three-dimensional morphology and spatial information of organs. This technique overcomes the limitations of traditional two-dimensional images in visually representing complex anatomical structures, and can accurately display organ information. However, current technology for three-dimensional organ reconstruction tends to result in low accuracy. Summary of the Invention

[0003] This application provides a method and apparatus for path planning based on multimodal data, a surgical robot, and an electronic device to improve the accuracy of three-dimensional reconstruction results of target organs.

[0004] Firstly, a method for planning paths based on multimodal data is provided, the method comprising: Acquire a first computed tomography (CT) image, the first CT image including the target organ, the first CT image not including contrast information, the contrast information being generated in the presence of a contrast agent; A first model is obtained, which is trained based on a loss function. The loss function is used to determine the difference between first information and second information. The first information is information obtained by the first model based on a second CT image, and the second information is information based on a third CT image. The second CT image does not include the angiography information, and the third CT image includes the angiography information. Both the first information and the second information include information related to the target organ. The first CT image is input into the first model to obtain third information related to the target organ, the third information including the imaging information; Based on the third information, the target organ is reconstructed in three dimensions to obtain the first reconstruction result.

[0005] In conjunction with any embodiment of this application, the method further includes: Acquire at least one first ultrasound image, said at least one first ultrasound image being acquired by an ultrasound acquisition device, the first ultrasound image including the target organ; Based on the at least one first ultrasound image, the target organ is reconstructed in three dimensions to obtain a second reconstruction result, the second reconstruction result including the three-dimensional information of the target organ in a first coordinate system; By registering the first reconstruction result and the second reconstruction result, the first reconstruction result is transformed into the first coordinate system to obtain the third reconstruction result.

[0006] In any embodiment of this application, the target organ is an organ of a target object, the target organ includes a target point, and the method further includes: Based on the first reconstruction result, determine the first endpoint corresponding to the target point; Determine the second endpoint corresponding to the first endpoint from the third reconstruction result; Acquire a second ultrasound image, which is acquired by the ultrasound acquisition device after acquiring the at least one first ultrasound image; A first starting point is determined from the skin region of the target object based on the second ultrasound image; Based on the first starting point and the second ending point, a target path in the first coordinate system is obtained, wherein the target path is the path from the skin of the target object to the target point.

[0007] In conjunction with any embodiment of this application, before determining the first starting point from the skin region of the target object based on the second ultrasound image, the method further includes: Determine the first position of the second ultrasound image in the first coordinate system; A fourth CT image is determined from the cross-section of the third reconstruction result based on the first position, and the position of the fourth CT image in the first coordinate system is the first position; Determine the first contour of the target organ in the fourth CT image; Determine the second contour of the target organ in the second ultrasound image; Determine the first similarity between the first contour and the second contour; Determining the first starting point from the skin region of the target object based on the second ultrasound image includes: If the first similarity is greater than or equal to the first threshold, the first starting point is determined from the skin region of the target object based on the second ultrasound image.

[0008] In conjunction with any embodiment of this application, determining the first similarity between the first contour and the second contour includes: Determine the intersection region between the area enclosed by the first contour and the area enclosed by the second contour; Determine the first area of ​​the intersection region; Determine the second area of ​​the region enclosed by the first contour; The first similarity is obtained based on the ratio of the first area to the second area, and the ratio is positively correlated with the first similarity.

[0009] In any embodiment of this application, the at least one first ultrasound image includes a third ultrasound image and a fourth ultrasound image. The third ultrasound image is obtained by acquiring the image of the target organ from a first perspective, and the fourth ultrasound image is obtained by acquiring the image of the target organ from a second perspective. The first perspective and the second perspective are different. The step of performing three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result includes: Based on the third ultrasound image and the fourth ultrasound image, at least one pair of first similar pixels is obtained. The first similar pixel pair includes a pixel obtained based on the third ultrasound image and a pixel obtained based on the fourth ultrasound image, and the similarity between the two pixels in the first similar pixel pair is greater than or equal to a second threshold. Based on the at least one pair of first similar pixels, the target organ is reconstructed in three dimensions to obtain the second reconstruction result.

[0010] In conjunction with any embodiment of this application, the second information is used to determine at least one of the following: the outline of the target organ, the tissue in the target organ, the topological relationship of the tissue in the target organ, and the correspondence between pixels in the second CT image and the tissue in the target organ.

[0011] Secondly, an apparatus for planning paths based on multimodal data is provided, the apparatus comprising: An acquisition unit is configured to acquire a first CT image, the first CT image including a target organ, the first CT image not including contrast information, the contrast information being generated in the presence of a contrast agent; The acquisition unit is further configured to acquire a first model, which is obtained by training based on a loss function. The loss function is used to determine the difference between the first information and the second information. The first information is information obtained by the first model based on a second CT image, and the second information is information obtained based on a third CT image. The second CT image does not include the angiography information, and the third CT image includes the angiography information. Both the first information and the second information include information related to the target organ. A processing unit is configured to input the first CT image into the first model to obtain third information related to the target organ, the third information including the imaging information; The processing unit is further configured to perform three-dimensional reconstruction of the target organ based on the third information to obtain a first reconstruction result.

[0012] In conjunction with any embodiment of this application, the acquisition unit is further configured to acquire at least one first ultrasound image, wherein the at least one first ultrasound image is acquired by an ultrasound acquisition device, and the first ultrasound image includes the target organ; The processing unit is further configured to perform three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result, wherein the second reconstruction result includes the three-dimensional information of the target organ in a first coordinate system; The processing unit is further configured to register the first reconstruction result and the second reconstruction result to transform the first reconstruction result into the first coordinate system to obtain a third reconstruction result.

[0013] In any embodiment of this application, the target organ is the organ of the target object, the target organ includes a target point, and the processing unit is further configured to determine a first endpoint corresponding to the target point based on the first reconstruction result; The processing unit is further configured to determine a second endpoint corresponding to the first endpoint from the third reconstruction result; The acquisition unit is further configured to acquire a second ultrasound image, which is acquired by the ultrasound acquisition device after acquiring the at least one first ultrasound image; The processing unit is further configured to determine a first starting point from the skin region of the target object based on the second ultrasound image; The processing unit is further configured to obtain a target path in the first coordinate system based on the first starting point and the second ending point, wherein the target path is a path from the skin of the target object to the target point.

[0014] In conjunction with any embodiment of this application, the processing unit is further configured to determine a first position of the second ultrasound image in the first coordinate system; The processing unit is further configured to determine a fourth CT image from the cross-section of the third reconstruction result based on the first position, wherein the position of the fourth CT image in the first coordinate system is the first position; The processing unit is further configured to determine a first contour of the target organ in the fourth CT image; The processing unit is further configured to determine a second contour of the target organ in the second ultrasound image; The processing unit is further configured to determine a first similarity between the first contour and the second contour; The processing unit is further configured to determine the first starting point from the skin region of the target object based on the second ultrasound image if the first similarity is greater than or equal to the first threshold.

[0015] In conjunction with any embodiment of this application, the processing unit is further configured to: Determine the intersection region between the area enclosed by the first contour and the area enclosed by the second contour; Determine the first area of ​​the intersection region; Determine the second area of ​​the region enclosed by the first contour; The first similarity is obtained based on the ratio of the first area to the second area, and the ratio is positively correlated with the first similarity.

[0016] In any embodiment of this application, the at least one first ultrasound image includes a third ultrasound image and a fourth ultrasound image. The third ultrasound image is obtained by acquiring the image of the target organ from a first perspective, and the fourth ultrasound image is obtained by acquiring the image of the target organ from a second perspective. The first perspective and the second perspective are different. The processing unit is further configured to: Based on the third ultrasound image and the fourth ultrasound image, at least one pair of first similar pixels is obtained. The first similar pixel pair includes a pixel obtained based on the third ultrasound image and a pixel obtained based on the fourth ultrasound image, and the similarity between the two pixels in the first similar pixel pair is greater than or equal to a second threshold. Based on the at least one pair of first similar pixels, the target organ is reconstructed in three dimensions to obtain the second reconstruction result.

[0017] In conjunction with any embodiment of this application, the second information is used to determine at least one of the following: the outline of the target organ, the tissue in the target organ, the topological relationship of the tissue in the target organ, and the correspondence between pixels in the second CT image and the tissue in the target organ.

[0018] Thirdly, a surgical robot is provided, including a device for multimodal data-based path planning as described in the second aspect. In this third aspect, the surgical robot can execute a multimodal data-based path planning method using the device, thereby improving the accuracy of the three-dimensional reconstruction results of the target organ.

[0019] Fourthly, an electronic device is provided, comprising: a processor and a memory, the memory for storing computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0020] Fifthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0021] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method as described in the first aspect above and any possible implementation thereof.

[0022] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, wherein, when the computer program or instructions are executed on a computer, the computer performs the method described in the first aspect and any possible implementation thereof.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0024] In this embodiment, because the first model has the ability to obtain organ information including contrast information from an organ CT image that does not include contrast information, the planning device, after acquiring the first CT image, can obtain organ information including contrast information, i.e., the third information, by inputting the first CT image into the first model. This enables the acquisition of third information including contrast information from a first CT image that does not include contrast information. Because the third information includes not only contrast information but also information related to the target organ in the first CT image, the planning device can then perform three-dimensional reconstruction of the target organ based on the third information to obtain the first reconstruction result, thereby improving the accuracy of the first reconstruction result. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be described below.

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0027] Figure 1 A flowchart illustrating a method for path planning based on multimodal data, provided in an embodiment of this application; Figure 2 A flowchart illustrating a training method provided in an embodiment of this application; Figure 3 A schematic diagram of a three-dimensional reconstruction of a kidney provided for an embodiment of this application; Figure 4 A schematic diagram illustrating a process for obtaining a fusion result, provided as an embodiment of this application; Figure 5 A schematic diagram illustrating the connection relationship between a guide, a positioning bracket, and an ultrasonic probe, provided in an embodiment of this application; Figure 6 A comparative schematic diagram of a first contour and a second contour provided for embodiments of this application; Figure 7 A comparative schematic diagram of another first profile and second profile provided for an embodiment of this application; Figure 8 A flowchart illustrating a multimodal data navigation method provided in this application embodiment; Figure 9 A schematic diagram of a device for planning paths based on multimodal data, provided in an embodiment of this application; Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. It should be understood that in this application, "at least one" means one or more, "more" means two or more, and "at least two" means two or three or more.

[0030] The execution subject of this application embodiment is a device for planning paths based on multimodal data (hereinafter referred to as the planning device), wherein the planning device can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the planning device can be one of the following: a computer, a server.

[0031] Before introducing the technical solutions of the embodiments of this application, the technical terms in the embodiments of this application will be explained.

[0032] Contrast agents (also known as contrast agents) are chemical preparations injected into target tissues to enhance the visual effect of imaging. These preparations have a density difference compared to the surrounding uninjected tissue; this density difference creates a clear image, making the injected tissue stand out more in the image and thus facilitating observation. For example, after injecting a contrast agent into a human kidney, an ultrasound image of the kidney taken using an ultrasound probe will clearly show the blood vessels filled with contrast agent in the kidney.

[0033] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a method for path planning based on multimodal data, provided in an embodiment of this application.

[0034] 101. Obtain the first CT image.

[0035] In this embodiment, the first CT image includes a target organ, which can be any organ. Optionally, the target organ includes at least one of the following: kidney, lung. The first CT image is obtained by a CT scanning device performing a CT scan on the target organ.

[0036] The first CT image does not include contrast information, which is generated in the presence of a contrast agent. Optionally, because the contrast agent can enhance the information in the CT image, the contrast information includes information enhanced by the contrast agent. For example, if the target organ is the kidney, without contrast agent, a CT image obtained by scanning the kidney will show the kidney's outline and kidney stones, but will lack information related to blood vessels and the renal collecting system. However, if contrast agent is present in the kidney, the CT image obtained by scanning the kidney will not only show the kidney's outline but also include information related to blood vessels and the renal collecting system; that is, the contrast information includes information related to blood vessels and the renal collecting system.

[0037] In one alternative implementation, the first CT image is acquired during the plain scan phase, during which the target organ is not exposed to contrast agent.

[0038] 102. Obtain the first model.

[0039] In this embodiment, the first model is trained based on a loss function, which is used to determine the difference between the first information and the second information. The first information is information obtained by the first model based on a second CT image, and the second information is information obtained based on a third CT image. The second CT image does not include contrast information, while the third CT image does include contrast information. Both the first and second information include information related to the target organ. That is, the first model has the ability to obtain organ information including contrast information based on organ CT images that do not include contrast information, where the organ CT image is a CT image including the target organ, and the organ information is information related to the target organ.

[0040] In one optional implementation, the training process of the first model includes: inputting a second CT image into the first model, so that the first model obtains information related to the target organ based on the second CT image, thus obtaining first information. Since the second CT image does not contain contrast information, the first information is organ information extracted from the organ CT image without contrast information. Second information related to the target organ is obtained based on a third CT image. Since the third CT image includes contrast information, the second information is organ information including contrast information. The difference between the first information and the second information is then calculated based on a loss function to obtain a loss function value, and the parameters of the first model are updated based on the loss function value so that the loss function value is less than or equal to a third threshold. This reduces the difference between the information obtained by the first model based on the organ CT image without contrast information and the second information, thereby enabling the first model to obtain organ information including contrast information based on the organ CT image without contrast information.

[0041] Optionally, the second information is obtained by the second model based on the third CT image, wherein the second model has the ability to extract organ information including contrast information from organ CT images that include contrast information. In this case, the first model can be trained using knowledge distillation, so that the first model is trained under the supervision of the second model, enabling the first model to obtain organ information including contrast information based on organ CT images that do not include contrast information.

[0042] In one possible implementation, the second information is used to determine at least one of the following: the outline of the target organ, the tissue in the target organ, the topological relationship of the tissue in the target organ, and the correspondence between pixels in the second CT image and the tissue in the target organ.

[0043] For example, the target organ is the kidney, and the tissues of the target organ include the blood vessels and the renal collecting system. Optionally, determining the tissues of the target organ includes determining the correspondence between image regions in the second CT image and the tissues of the target organ. For example, the correspondence between image regions in the second CT image and the tissues of the target organ may include: which image regions in the second CT image correspond to the blood vessels in the kidney, and which image regions in the second CT image correspond to the renal collecting system.

[0044] The topological relationships of tissues within a target organ include at least one of the following: the connectivity relationships of tissues within the target organ, and the relative positional relationships of tissues within the target organ. For example, if the target organ is the kidney, the topological relationships of the tissues within the target organ include the connectivity relationships of blood vessels within the kidney, and the relative positional relationships between the renal collecting system and the blood vessels within the kidney.

[0045] Determining the correspondence between pixels in the second CT image and tissues in the target organ is equivalent to determining the semantic meaning of the pixels in the second CT image. Based on this correspondence, the second CT image can be segmented.

[0046] Optionally, the second and third CT images are acquired in the same imaging process, wherein an imaging process includes a period of time before the contrast agent enters the target organ, the contrast agent enters the target organ, and the contrast agent flows out of the target organ. For example, the period of time is 20 seconds.

[0047] Optionally, the target organ is the kidney, and the third CT image is acquired during the cortical phase or the excretory phase. In both the cortical and excretory phases, the kidney contains contrast agent.

[0048] In one alternative implementation, Figure 2 This is a flowchart illustrating a training method provided in an embodiment of this application. Based on Figure 2 The training method shown trains the first model, enabling it to obtain organ information including contrast information from organ CT images that do not include contrast information.

[0049] exist Figure 2 In the training method shown, the target organ is the kidney. First, the second CT image, the third CT image, and the excretory phase CT image are cropped to obtain the image corresponding to the kidney. The second CT image was acquired during the plain scan phase, the third CT image was acquired during the cortical phase, and the excretory phase CT image was acquired during the excretory phase.

[0050] Optionally, the second CT image, the third CT image, and the excretion-phase CT image are all three-dimensional images. Optionally, the cropping can be cropping the image corresponding to the left kidney or cropping the image corresponding to the right kidney. For example, the size of the cropped image corresponding to the kidney is 128×128×128 or 256×512×128.

[0051] The first cropped image is obtained by cropping the second CT image. The second cropped image is obtained by cropping the third CT image. The third cropped image is obtained by cropping the CT image from the excretory period. The first, second, and third cropped images are registered so that the second and third cropped images are aligned with the first cropped image, resulting in aligned second and third cropped images. Since the respiratory state of the subject varies at different times, different respiratory states can lead to greater differences between organ CT images acquired at different times. Therefore, this registration can reduce the influence of respiratory state, thereby helping to reduce errors in subsequent processing. The first cropped image, the aligned second cropped image, and the aligned third cropped image are fused to obtain the fused image.

[0052] The first cropped image is input to the encoder of the first model, which extracts features from the first cropped image to obtain first feature data. The fused image is input to the encoder of the second model, which extracts features from the fused image to obtain second feature data and topological relationships, where the topological relationships are the topological relationships of the tissues in the kidney. Optionally, the encoder of the second model obtains a spatial attention weight map based on the features of blood vessels and renal calyces in the kidney, where the spatial attention weight map includes the topological relationships between blood vessels and renal calyces in the kidney. Based on the spatial attention weight map, the topological relationships of the tissues in the kidney are obtained. The decoder of the first model then obtains the segmentation result of the first cropped image, i.e., the first segmentation result, based on the first feature data, where the first segmentation result indicates the semantics of the pixels in the first cropped image. The decoder of the second model then obtains the segmentation result of the fused image, i.e., the second segmentation result, based on the second feature data, where the second segmentation result indicates the semantics of the pixels in the fused image. At this point, the first information includes the first feature data and the first segmentation result, and the second information includes the second feature data, topological relationships, and the second segmentation result.

[0053] The difference between the first and second information is then determined based on the loss function to obtain the loss function value. Specifically, a first loss value is obtained based on the difference between the first feature data and the topological relationship, where the greater the difference between the first feature data and the topological relationship, the larger the first loss value. A second loss value is obtained based on the difference between the first and second feature data, where the greater the difference between the first and second feature data, the larger the second loss value. Optionally, a second loss value is obtained by determining the Kullback-Leibler (KL) divergence between the first and second feature data. A third loss value is obtained based on the difference between the first and second segmentation results, where the greater the difference between the first and second segmentation results, the larger the third loss value. The first, second, and third loss values ​​are then weighted and summed to obtain the loss function value.

[0054] Optionally, the first feature data is represented as The second feature data is represented as Topological relationships are represented as The first segmentation result is represented as The second segmentation result is represented as The loss function value is expressed as .So , , , , , Satisfy the following formula:

[0055] in, , , All are weights. This represents the first loss value. This represents the second loss value. This represents the third loss value.

[0056] Finally, based on the loss function value, adjust the parameters of the first model until the loss function value is less than or equal to the third threshold, and then stop training.

[0057] 103. Input the first CT image into the first model to obtain third information related to the target organ.

[0058] Because the first model has the ability to obtain organ information including contrast information based on organ CT images that do not include contrast information, the planning device inputs the first CT image into the first model, which enables the first model to obtain third information related to the target organ based on the first CT image, and the third information includes contrast information.

[0059] Optionally, the first model predicts the contrast information in the third information based on information related to the target organ in the first CT image.

[0060] Optionally, the third information may also include information related to the target organ in the first CT image. For example, if the first CT image was acquired during the plain scan period, then the third information includes not only information related to the target organ during the plain scan period, but also contrast information.

[0061] 104. Based on the third information, the target organ is reconstructed in three dimensions to obtain the first reconstruction result.

[0062] Because the third information includes information related to the target organ, the planning device can perform three-dimensional reconstruction of the target organ based on the third information to obtain the first reconstruction result.

[0063] Optionally, the first reconstruction result includes three-dimensional information of the target organ. For example, the three-dimensional information includes the three-dimensional dimensions of the target organ and the three-dimensional position information of the target organ.

[0064] exist Figure 1 In this method, because the first model has the ability to obtain organ information including contrast information from organ CT images that do not include contrast information, the planning device, after acquiring the first CT image, can obtain organ information including contrast information, i.e., the third information, by inputting the first CT image into the first model. This enables the acquisition of third information including contrast information from a first CT image that does not include contrast information. Because the third information includes not only contrast information but also information related to the target organ in the first CT image, the planning device can then perform three-dimensional reconstruction of the target organ based on the third information to obtain the first reconstruction result, thereby improving the accuracy of the first reconstruction result.

[0065] In one possible scenario, if the target subject is allergic to contrast agents, it is difficult to acquire CT images containing contrast information. In this case, if 3D reconstruction of the target organ is required, a method can be used... Figure 1 This method allows for three-dimensional reconstruction of the target organ, thereby improving the accuracy of the reconstruction results. Specifically, in... Figure 1 In this method, the planning device can obtain third information including contrast information from the target organ based on CT images that do not include contrast information, and then perform three-dimensional reconstruction of the target organ based on the third information, which can improve the accuracy of the reconstruction results.

[0066] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a three-dimensional reconstruction of a kidney, provided as an embodiment of this application. Figure 3As shown, the process first determines whether the CT image includes contrast information. If it does, a 3D reconstruction of the kidney can be performed based on the first CT image, the cortical phase CT image, and the excretory phase CT image. The first CT image was acquired during the plain scan phase, and the cortical phase CT image was acquired during the cortical phase. Specifically, based on the plain scan phase segmentation algorithm, the first CT image is segmented to obtain first segmentation information, which includes the following information: kidney stones and the outline of the kidney. Based on the cortical phase segmentation algorithm, the cortical phase CT image is segmented to obtain second segmentation information, which includes the following information: blood vessels in the kidney, the renal medulla, and the outline of the kidney. Based on the excretory phase segmentation algorithm, the excretory phase CT image is segmented to obtain third segmentation information, which includes the following information: the renal collecting system and the outline of the kidney. Then, a 3D reconstruction of the kidney is performed based on the first, second, and third segmentation information. If the CT image does not include contrast information, such as if the target subject is allergic to contrast agents, a 3D reconstruction of the kidney can be performed based on... Figure 1 The method processes the first CT image to obtain third information, which includes first segmentation information, second segmentation information, and third segmentation information. Then, based on the third information, a three-dimensional reconstruction of the kidney is performed.

[0067] As an optional implementation, the planning device further performs the following steps: acquiring at least one first ultrasound image, wherein the at least one first ultrasound image is acquired by an ultrasound acquisition device, and the first ultrasound image includes the target organ; performing three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result, wherein the second reconstruction result includes three-dimensional information of the target organ in a first coordinate system; and registering the first reconstruction result and the second reconstruction result to transform the first reconstruction result to the first coordinate system to obtain a third reconstruction result. Optionally, the first coordinate system is the world coordinate system.

[0068] The planning device, based on this implementation, can align the 3D reconstruction results obtained from CT images and the 3D reconstruction results obtained from ultrasound images. In one possible scenario, after obtaining the third reconstruction result, the first and third reconstruction results are fused to obtain a fused result, which is then displayed. This allows the user to determine the information of the target organ based on the fused result. For example, CT images can provide more information related to the target organ than ultrasound images, while ultrasound images offer better real-time performance. Therefore, by aligning the 3D reconstruction results obtained from CT images and the 3D reconstruction results obtained from ultrasound images, the information related to the target organ in the CT images can be used to compensate for the lack of relevant information in the ultrasound images, while also utilizing the real-time information of the ultrasound images. This allows the user to better determine the information of the target organ.

[0069] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating a process for obtaining a fusion result, provided as an embodiment of this application. Figure 4 As shown, a first reconstruction result can be obtained through CT reconstruction. Specifically, a CT scan is performed to obtain a first CT image. Then, three-dimensional reconstruction is performed based on the first CT image to obtain an intermediate reconstruction result. The image region corresponding to the kidney is then cropped from the intermediate reconstruction result to obtain the first reconstruction result. A second reconstruction result can be obtained based on an optically positioned ultrasound probe and ultrasound-guided kidney positioning. Specifically, for ultrasound-guided kidney positioning, at least one first ultrasound image including the kidney can be acquired using an ultrasound probe. The optically positioned ultrasound probe can determine the position of the kidney in the world coordinate system within the at least one first ultrasound image. Based on this position, three-dimensional reconstruction of the kidney can be performed to obtain the second reconstruction result. After obtaining the third reconstruction result based on the first reconstruction result, the second and third reconstruction results are fused to obtain the fused result.

[0070] The ultrasonic probe is connected to a guide, and the movement of the guide drives the movement of the ultrasonic probe. The ultrasonic probe is also connected to a positioning bracket, which includes four optical marker balls. An optical tracking device can locate the ultrasonic probe using optical positioning technology by tracking these four optical marker balls. For example... Figure 5 This is a schematic diagram illustrating the connection relationship between a guide, a positioning bracket, and an ultrasonic probe, provided in an embodiment of this application.

[0071] As an optional implementation, the target organ is the organ of the target object, and the target organ includes a target point. The planning device further performs the following steps: determining a first endpoint corresponding to the target point based on a first reconstruction result; determining a second endpoint corresponding to the first endpoint from a third reconstruction result; acquiring a second ultrasound image, wherein the second ultrasound image is acquired by an ultrasound acquisition device after acquiring at least one first ultrasound image; determining a first starting point from the skin region of the target object based on the second ultrasound image; and obtaining a target path in a first coordinate system based on the first starting point and the second endpoint, wherein the target path is the path from the skin of the target object to the target point.

[0072] On the one hand, because CT images contain more information related to the target organ than ultrasound images, and the first reconstruction result is based on the first CT image, when it is necessary to determine the path from the skin of the target object to the target point, the first endpoint corresponding to the target point in the target organ can be determined based on the first reconstruction result. On the other hand, because the ultrasound acquisition device can acquire a second ultrasound image including the target organ by moving on the skin of the target object, the first starting point can be determined based on the second ultrasound image. After determining the first endpoint, the planning device further determines the second endpoint corresponding to the first endpoint from the third reconstruction result, thereby determining the point corresponding to the target point in the first coordinate system. Finally, based on the first starting point and the second endpoint, the target path is obtained, which can improve the accuracy of the target path, wherein the second endpoint is the endpoint of the target path.

[0073] Optionally, the first starting point is a point in a first coordinate system. After determining the second starting point of the target path from the second ultrasound image, the planning device can obtain the first starting point based on the first transformation relationship and the position of the second starting point in the pixel coordinate system of the second ultrasound image, wherein the first transformation relationship is the transformation relationship between the position in the pixel coordinate system and the position in the first coordinate system.

[0074] In one possible scenario, the acquisition time of the first CT image is referred to as the CT acquisition time, and the acquisition time of the second ultrasound image is referred to as the ultrasound acquisition time, wherein the CT acquisition time is earlier than the ultrasound acquisition time. After the planning device acquires the second ultrasound image, the target path can be planned. Subsequently, the relevant device can be navigated based on the target path. For example, the relevant device is a guide, wherein the guide includes a placement hole for placing a moving object, wherein when the moving object is placed in the placement hole, the moving object coincides with the axis of the placement hole. By navigating the guide based on the target path, the axis of the placement hole can be made to coincide with the target path, thereby moving the moving object along the axis of the placement hole, thus moving the moving object along the target path.

[0075] As an optional implementation, before the planning device performs the step "determine the first starting point from the skin region of the target object based on the second ultrasound image", it further performs the following steps: determining a first position of the second ultrasound image in a first coordinate system; determining a fourth CT image from the cross-section of the third reconstruction result based on the first position, wherein the position of the fourth CT image in the first coordinate system is the first position; determining a first contour of the target organ in the fourth CT image; determining a second contour of the target organ in the second ultrasound image; and determining a first similarity between the first contour and the second contour. After obtaining the first similarity, the following step may be performed during the execution of the step "determine the first starting point from the skin region of the target object based on the second ultrasound image": if the first similarity is greater than or equal to a first threshold, determining the first starting point from the skin region of the target object based on the second ultrasound image.

[0076] Considering that the respiratory state of the target object varies at different times, and that different respiratory states lead to different states of the target organ, which in turn leads to different locations of the target point within the target organ. Optionally, the state of the target organ includes the shape and / or location of the target organ. For example, the target organ is the kidney, and the target object's breathing will cause changes in the shape and / or location of the kidney.

[0077] Because the CT and ultrasound acquisition times differ, the target's position at the CT acquisition time may differ from its position at the ultrasound acquisition time. When the target's position differs between the CT and ultrasound acquisition times, determining the first starting point based on the second ultrasound image and obtaining the target path in the first coordinate system based on the first starting point and the second ending point can easily lead to a large deviation between the target path and the target; in other words, it can easily result in a large distance between the target path and the target.

[0078] Based on this, the planning device first determines the first position of the second ultrasound image in the first coordinate system, and then determines the CT cross-section at the first position based on the third reconstruction result, thus obtaining the fourth CT image. On one hand, the third reconstruction result is based on the first reconstruction result, which in turn is based on the first CT image; therefore, the state of the target organ in the fourth CT image is the state of the target organ at the time of CT acquisition. On the other hand, the state of the target organ in the second ultrasound image is the state of the target organ at the time of ultrasound acquisition. Therefore, the difference between the state of the target organ at the time of CT acquisition and the state of the target organ at the time of ultrasound acquisition can be determined based on the similarity between the state of the target organ in the fourth CT image and the state of the target organ in the second ultrasound image.

[0079] Therefore, after determining the first contour and the second contour, the planning device determines the first similarity between the first contour and the second contour. Based on the magnitude of the first similarity, it can determine whether the difference between the state of the target organ at the time of CT acquisition and the state of the target organ at the time of ultrasound acquisition is large or small. Specifically, a large first similarity indicates a large difference, and a small first similarity indicates a small difference. In this embodiment, the planning device determines whether the first similarity is large or small based on a first threshold. Specifically, a first similarity greater than or equal to the first threshold indicates a large first similarity. Conversely, a first similarity less than the first threshold indicates a small first similarity.

[0080] Therefore, when the first similarity is greater than or equal to the first threshold, the planning device determines the first starting point from the skin region of the target object based on the second ultrasound image. This can be achieved when the difference between the state of the target organ at the time of CT acquisition and the state of the target organ at the time of ultrasound acquisition is small. Subsequently, the target path is obtained based on the second endpoint and the first starting point, which can reduce the deviation between the target path and the target point.

[0081] Please see Figure 6 , Figure 6 This is a comparative schematic diagram of a first contour and a second contour provided for embodiments of this application. For example... Figure 6 As shown, the first contour and the second contour have a high degree of similarity, at which point a first starting point can be determined from the skin region of the target object based on the second ultrasound image. See also... Figure 7 , Figure 7 This is a comparative schematic diagram of another first profile and a second profile provided for an embodiment of this application. For example... Figure 7 As shown, the first contour and the second contour have a small first similarity, at which point the planning device does not determine the first starting point from the skin region of the target object based on the second ultrasound image.

[0082] As an optional implementation, determining the first similarity between the first contour and the second contour includes the following steps: determining the intersection region of the region enclosed by the first contour and the region enclosed by the second contour; determining the first area of ​​the intersection region; determining the second area of ​​the region enclosed by the first contour; and obtaining a first similarity based on a first ratio of the first area to the second area, wherein the first ratio is positively correlated with the first similarity.

[0083] In this embodiment, the larger the first ratio of the first area to the second area, the greater the overlap between the first contour and the second contour, and thus the greater the similarity between the first contour and the second contour. Therefore, the planning device can improve the accuracy of the first similarity by obtaining the first similarity based on this embodiment.

[0084] As an optional implementation, determining a first similarity between a first contour and a second contour includes the following steps: Based on at least one pair of second similar pixel pairs between the first and second contours, wherein each second similar pixel pair includes a pixel in the first contour and a pixel in the second contour, and the semantic similarity of the two pixels in the second similar pixel pair is greater than or equal to a fourth threshold. Determining a first number of second similar pixel pairs in the at least one pair. Determining a second number of pixels in the first contour. Determining a second ratio between the first number and the second number. Determining the distance between the two pixels in each second similar pixel pair in the at least one pair, obtaining at least one first distance. Based on the at least one first distance, obtaining a second distance, wherein the second distance is one of the following: the average of at least one first distance, or the maximum value of at least one first distance. Based on the second ratio and the second distance, obtaining a first similarity, wherein the second ratio is positively correlated with the first similarity, and the second distance is negatively correlated with the first similarity.

[0085] In this implementation, a larger second ratio indicates a greater semantic similarity between the first contour and the second contour. A larger second distance indicates a greater positional difference between the first contour and the second contour. Therefore, the planning device, based on the principle that the second ratio is positively correlated with the first similarity and the second distance is negatively correlated with the first similarity, obtains the first similarity based on the second ratio and the second distance, thereby improving the accuracy of the first similarity.

[0086] As an optional implementation, at least one first ultrasound image includes a third ultrasound image and a fourth ultrasound image. The third ultrasound image is acquired from a first perspective of the target organ, and the fourth ultrasound image is acquired from a second perspective of the target organ. The first and second perspectives are different. In other words, the third and fourth ultrasound images are acquired from different angles of the target organ. Therefore, the organ information in the third ultrasound image and the organ information in the fourth ultrasound image are organ information observed from different angles.

[0087] In this embodiment, a second reconstruction result is obtained by performing three-dimensional reconstruction of the target organ based on at least one first ultrasound image, including the following steps: obtaining at least one pair of first similar pixel pairs based on a third ultrasound image and a fourth ultrasound image, wherein the first similar pixel pair includes a pixel obtained based on the third ultrasound image and a pixel obtained based on the fourth ultrasound image, and the similarity between the two pixels in the first similar pixel pair is greater than or equal to a second threshold. The second reconstruction result is obtained by performing three-dimensional reconstruction of the target organ based on at least one pair of first similar pixel pairs.

[0088] Because the first similar pixel pair includes one pixel obtained from the third ultrasound image and one pixel obtained from the fourth ultrasound image, the two pixels in the first similar pixel pair carry organ information observed from different angles. Furthermore, because the similarity between the two pixels in the first similar pixel pair is greater than or equal to the second threshold, it indicates that the organ information corresponding to the first similar pixel pair (i.e., the information carried by the pixels in the first similar pixel pair) can be observed from different angles, which also indicates that the accuracy of this information is high. Therefore, the planning device performs three-dimensional reconstruction of the target organ based on at least one pair of first similar pixel pairs to obtain the second reconstruction result, which can improve the accuracy of the second reconstruction result.

[0089] As an optional implementation, obtaining at least one pair of first similar pixel pairs based on a third ultrasound image and a fourth ultrasound image includes the following steps: performing three-dimensional reconstruction of the target organ based on images from the third ultrasound image set to obtain a fourth reconstruction result; performing three-dimensional reconstruction of the target organ based on images from the fourth ultrasound image set to obtain a fifth reconstruction result; and obtaining at least one pair of first similar pixel pairs based on the fourth and fifth reconstruction results, wherein the first similar pixel pair includes a pixel from the fourth reconstruction result and a pixel from the fifth reconstruction result.

[0090] In this implementation, since the three-dimensional reconstruction results can more comprehensively reflect the information of the target organ, at least one pair of first similar pixels is obtained based on the fourth and fifth reconstruction results, which can improve the accuracy of at least one pair of first similar pixels.

[0091] Please see Figure 8 , Figure 8 This is a schematic diagram illustrating a multimodal data-based navigation process provided in an embodiment of this application. Figure 8 First, ultrasound calibration is performed. Specifically, ultrasound calibration determines the transformation relationship (i.e., the first transformation relationship) between the pixel coordinate system of the ultrasound image acquired by the ultrasound acquisition device and the first coordinate system. Then, the target organ is determined; for example, the target organ is determined to be the left or right kidney. A first reconstruction result is then obtained based on the first CT image. Based on the first reconstruction result, a first endpoint is obtained. A second reconstruction result is obtained based on at least one first ultrasound image; specifically, the second reconstruction result is obtained based on at least one first ultrasound image and the ultrasound calibration result. A starting point is obtained based on the second reconstruction result. Based on the starting point and the first endpoint, a target path is obtained; specifically, a second endpoint is first obtained based on the first endpoint, and then the target path is obtained based on the second endpoint and the starting point. Navigation of relevant devices is performed based on the target path.

[0092] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0093] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0094] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0095] Please see Figure 9 , Figure 9 This is a schematic diagram of a device for path planning based on multimodal data, provided in an embodiment of this application. The device 1 for path planning based on multimodal data includes: an acquisition unit 11 and a processing unit 12, wherein: Acquisition unit 11 is used to acquire a first CT image, the first CT image including the target organ, the first CT image not including contrast information, the contrast information being generated in the presence of contrast agent; The acquisition unit 11 is further configured to acquire a first model, which is obtained by training based on a loss function. The loss function is used to determine the difference between the first information and the second information. The first information is information obtained by the first model based on a second CT image. The second information is information obtained based on a third CT image. The second CT image does not include the angiography information. The third CT image includes the angiography information. Both the first information and the second information include information related to the target organ. Processing unit 12 is used to input the first CT image into the first model to obtain third information related to the target organ, the third information including the imaging information; The processing unit 12 is further configured to perform three-dimensional reconstruction of the target organ based on the third information to obtain a first reconstruction result.

[0096] In any embodiment of this application, the acquisition unit 11 is further configured to acquire at least one first ultrasound image, wherein the at least one first ultrasound image is acquired by an ultrasound acquisition device, and the first ultrasound image includes the target organ. The processing unit 12 is further configured to perform three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result, the second reconstruction result including the three-dimensional information of the target organ in a first coordinate system; The processing unit 12 is further configured to register the first reconstruction result and the second reconstruction result to transform the first reconstruction result into the first coordinate system to obtain a third reconstruction result.

[0097] In any embodiment of this application, the target organ is the organ of the target object, the target organ includes a target point, and the processing unit 12 is further configured to determine a first endpoint corresponding to the target point based on the first reconstruction result; The processing unit 12 is further configured to determine a second endpoint corresponding to the first endpoint from the third reconstruction result; The acquisition unit 11 is also used to acquire a second ultrasound image, which is acquired by the ultrasound acquisition device after acquiring the at least one first ultrasound image; The processing unit 12 is further configured to determine a first starting point from the skin region of the target object based on the second ultrasound image; The processing unit 12 is further configured to obtain a target path in the first coordinate system based on the first starting point and the second ending point, wherein the target path is a path from the skin of the target object to the target point.

[0098] In any embodiment of this application, the processing unit 12 is further configured to determine the first position of the second ultrasound image in the first coordinate system; The processing unit 12 is further configured to determine a fourth CT image from the cross section of the third reconstruction result based on the first position, wherein the position of the fourth CT image in the first coordinate system is the first position; The processing unit 12 is further configured to determine a first contour of the target organ in the fourth CT image; The processing unit 12 is further configured to determine a second contour of the target organ in the second ultrasound image; The processing unit 12 is further configured to determine a first similarity between the first contour and the second contour; The processing unit 12 is further configured to determine the first starting point from the skin region of the target object based on the second ultrasound image when the first similarity is greater than or equal to the first threshold.

[0099] In conjunction with any embodiment of this application, the processing unit 12 is further configured to: Determine the intersection region between the area enclosed by the first contour and the area enclosed by the second contour; Determine the first area of ​​the intersection region; Determine the second area of ​​the region enclosed by the first contour; The first similarity is obtained based on the ratio of the first area to the second area, and the ratio is positively correlated with the first similarity.

[0100] In any embodiment of this application, the at least one first ultrasound image includes a third ultrasound image and a fourth ultrasound image. The third ultrasound image is obtained by acquiring the image of the target organ from a first perspective, and the fourth ultrasound image is obtained by acquiring the image of the target organ from a second perspective. The first perspective and the second perspective are different. The processing unit 12 is further configured to: Based on the third ultrasound image and the fourth ultrasound image, at least one pair of first similar pixels is obtained. The first similar pixel pair includes a pixel obtained based on the third ultrasound image and a pixel obtained based on the fourth ultrasound image, and the similarity between the two pixels in the first similar pixel pair is greater than or equal to a second threshold. Based on the at least one pair of first similar pixels, the target organ is reconstructed in three dimensions to obtain the second reconstruction result.

[0101] In conjunction with any embodiment of this application, the second information is used to determine at least one of the following: the outline of the target organ, the tissue in the target organ, the topological relationship of the tissue in the target organ, and the correspondence between pixels in the second CT image and the tissue in the target organ.

[0102] In this embodiment, because the first model has the ability to obtain organ information including contrast information from an organ CT image that does not include contrast information, the planning device, after acquiring the first CT image, can obtain organ information including contrast information, i.e., the third information, by inputting the first CT image into the first model. This enables the acquisition of third information including contrast information from a first CT image that does not include contrast information. Because the third information includes not only contrast information but also information related to the target organ in the first CT image, the planning device can then perform three-dimensional reconstruction of the target organ based on the third information to obtain the first reconstruction result, thereby improving the accuracy of the first reconstruction result.

[0103] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0104] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0105] Processor 21 can be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0106] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0107] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0108] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. For example, the memory 22 can be used to store the first CT image and the first model obtained through the input device 23, or the memory 22 can also be used to store the first reconstruction result obtained through the processor 21, etc. This embodiment of the application does not limit the specific data stored in the memory.

[0109] Understandable Figure 10 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0110] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0115] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0116] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or RAM, magnetic disks, or optical disks.

Claims

1. A method for path planning based on multimodal data, characterized in that, The method includes: Acquire a first CT image, the first CT image including the target organ, the first CT image not including contrast information, the contrast information being generated in the presence of a contrast agent, the target organ being the organ of the target object, the target organ including the target point; A first model is obtained, which is trained based on a loss function. The loss function is used to determine the difference between first information and second information. The first information is information obtained by the first model based on a second CT image, and the second information is information based on a third CT image. The second CT image does not include the angiography information, and the third CT image includes the angiography information. Both the first information and the second information include information related to the target organ. The first CT image is input into the first model to obtain third information related to the target organ, the third information including the imaging information; Based on the third information, the target organ is reconstructed in three dimensions to obtain the first reconstruction result; Acquire at least one first ultrasound image, said at least one first ultrasound image being acquired by an ultrasound acquisition device, the first ultrasound image including the target organ; Based on the at least one first ultrasound image, the target organ is reconstructed in three dimensions to obtain a second reconstruction result, the second reconstruction result including the three-dimensional information of the target organ in a first coordinate system; The at least one first ultrasound image includes a third ultrasound image and a fourth ultrasound image. The third ultrasound image is obtained by acquiring the image of the target organ from a first perspective, and the fourth ultrasound image is obtained by acquiring the image of the target organ from a second perspective. The first perspective and the second perspective are different. The step of performing three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result includes: obtaining at least one pair of first similar pixel pairs based on the third ultrasound image and the fourth ultrasound image, wherein the similarity between the two pixels in the first similar pixel pair is greater than or equal to a second threshold; and performing three-dimensional reconstruction of the target organ based on the at least one pair of first similar pixel pairs to obtain the second reconstruction result. The step of obtaining at least one pair of first similar pixel pairs based on the third ultrasound image and the fourth ultrasound image includes: performing three-dimensional reconstruction of the target organ based on the third ultrasound image to obtain a fourth reconstruction result; performing three-dimensional reconstruction of the target organ based on the fourth ultrasound image to obtain a fifth reconstruction result; and obtaining the at least one pair of first similar pixel pairs based on the fourth reconstruction result and the fifth reconstruction result, wherein the first similar pixel pair includes a pixel from the fourth reconstruction result and a pixel from the fifth reconstruction result; By registering the first reconstruction result and the second reconstruction result, the first reconstruction result is transformed into the first coordinate system to obtain the third reconstruction result; Based on the first reconstruction result, determine the first endpoint corresponding to the target point; Determine the second endpoint corresponding to the first endpoint from the third reconstruction result; A second ultrasound image is acquired after the ultrasound acquisition device has acquired at least one first ultrasound image; the acquisition time of the second ultrasound image is different from the acquisition time of the first CT image. Determine the first position of the second ultrasound image in the first coordinate system; A fourth CT image is determined from the cross-section of the third reconstruction result based on the first position, and the position of the fourth CT image in the first coordinate system is the first position; Determine the first contour of the target organ in the fourth CT image; Determine the second contour of the target organ in the second ultrasound image; Determine the first similarity between the first contour and the second contour; If the first similarity is greater than or equal to the first threshold, a first starting point is determined from the skin region of the target object based on the second ultrasound image; Based on the first starting point and the second ending point, a target path in the first coordinate system is obtained, wherein the target path is the path from the skin of the target object to the target point.

2. The method according to claim 1, characterized in that, Determining the first similarity between the first contour and the second contour includes: Determine the intersection region between the area enclosed by the first contour and the area enclosed by the second contour; Determine the first area of ​​the intersection region; Determine the second area of ​​the region enclosed by the first contour; The first similarity is obtained based on the ratio of the first area to the second area, and the ratio is positively correlated with the first similarity.

3. The method according to claim 1 or 2, characterized in that, The second information is used to determine at least one of the following: the outline of the target organ, the tissue in the target organ, the topological relationship of the tissue in the target organ, and the correspondence between pixels in the second CT image and the tissue in the target organ.

4. A device for path planning based on multimodal data, characterized in that, The device includes: An acquisition unit is used to acquire a first CT image, the first CT image including a target organ, the first CT image not including contrast information, the contrast information being generated in the presence of a contrast agent, the target organ being the organ of the target object, the target organ including a target point; The acquisition unit is further configured to acquire a first model, which is obtained by training based on a loss function. The loss function is used to determine the difference between the first information and the second information. The first information is information obtained by the first model based on a second CT image, and the second information is information obtained based on a third CT image. The second CT image does not include the angiography information, and the third CT image includes the angiography information. Both the first information and the second information include information related to the target organ. A processing unit is configured to input the first CT image into the first model to obtain third information related to the target organ, the third information including the imaging information; The processing unit is further configured to perform three-dimensional reconstruction of the target organ based on the third information to obtain a first reconstruction result; The acquisition unit is further configured to acquire at least one first ultrasound image, which is acquired by an ultrasound acquisition device and includes the target organ. The processing unit is further configured to perform three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result, wherein the second reconstruction result includes the three-dimensional information of the target organ in a first coordinate system; The at least one first ultrasound image includes a third ultrasound image and a fourth ultrasound image. The third ultrasound image is obtained by acquiring the image of the target organ from a first perspective, and the fourth ultrasound image is obtained by acquiring the image of the target organ from a second perspective. The first perspective and the second perspective are different. The step of performing three-dimensional reconstruction of the target organ based on the at least one first ultrasound image to obtain a second reconstruction result includes: obtaining at least one pair of first similar pixel pairs based on the third ultrasound image and the fourth ultrasound image, wherein the similarity between the two pixels in the first similar pixel pair is greater than or equal to a second threshold; and performing three-dimensional reconstruction of the target organ based on the at least one pair of first similar pixel pairs to obtain the second reconstruction result. The step of obtaining at least one pair of first similar pixel pairs based on the third ultrasound image and the fourth ultrasound image includes: performing three-dimensional reconstruction of the target organ based on the third ultrasound image to obtain a fourth reconstruction result; performing three-dimensional reconstruction of the target organ based on the fourth ultrasound image to obtain a fifth reconstruction result; and obtaining the at least one pair of first similar pixel pairs based on the fourth reconstruction result and the fifth reconstruction result, wherein the first similar pixel pair includes a pixel from the fourth reconstruction result and a pixel from the fifth reconstruction result; The processing unit is further configured to register the first reconstruction result and the second reconstruction result to transform the first reconstruction result into the first coordinate system to obtain a third reconstruction result; The processing unit is further configured to determine a first endpoint corresponding to the target point based on the first reconstruction result; The processing unit is further configured to determine a second endpoint corresponding to the first endpoint from the third reconstruction result; The acquisition unit is further configured to acquire a second ultrasound image, which is acquired by the ultrasound acquisition device after acquiring the at least one first ultrasound image; the acquisition time of the second ultrasound image is different from the acquisition time of the first CT image. The processing unit is further configured to determine the first position of the second ultrasound image in the first coordinate system; The processing unit is further configured to determine a fourth CT image from the cross-section of the third reconstruction result based on the first position, wherein the position of the fourth CT image in the first coordinate system is the first position; The processing unit is further configured to determine a first contour of the target organ in the fourth CT image; The processing unit is further configured to determine a second contour of the target organ in the second ultrasound image; The processing unit is further configured to determine a first similarity between the first contour and the second contour; The processing unit is further configured to determine a first starting point from the skin region of the target object based on the second ultrasound image when the first similarity is greater than or equal to the first threshold. The processing unit is further configured to obtain a target path in the first coordinate system based on the first starting point and the second ending point, wherein the target path is a path from the skin of the target object to the target point.

5. A surgical robot, characterized in that, Includes the apparatus for planning paths based on multimodal data as described in claim 4.

6. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 3.

Images