Navigation method, system, computer device, storage medium and computer program product

Abstract

The application relates to a navigation method, system, computer device, storage medium and computer program product. The method comprises the following steps: obtaining a target medical model, the target medical model being reconstructed based on an initial image scanned by a scanning device; generating operation planning information based on the target medical model; obtaining a real-time image scanned by the scanning device, and registering the real-time image with the target medical model to obtain a first registration relationship; obtaining a first coordinate relationship between a coordinate system corresponding to an execution tool and an image coordinate system of the real-time image; and navigating the execution tool based on the first coordinate relationship, the first registration relationship and the operation planning information. The method can model during operation, does not require preoperative CT, and realizes real-time registration without the need of adding additional markers.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular to a navigation method, system, computer device, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, surgical navigation systems have emerged, aligning with the trend towards precision surgery. By analyzing patient medical images and utilizing various intraoperative sensors, surgical navigation systems provide richer reference information and more precise guidance for surgical procedures, becoming a powerful tool to assist surgeons in completing operations.

[0003] In traditional techniques, the preoperative surgical plan is prepared by collecting data such as the patient's CT scan before surgery, which is then completed in the computer of the surgical navigation system. The doctor then performs the surgery according to the preoperative surgical plan.

[0004] However, current surgical navigation systems require preoperative CT scans, which leads to a longer surgical preparation time. In addition, in order to register the bone with the CT scan, positioning markers need to be added to the target area, resulting in additional trauma. Summary of the Invention

[0005] Therefore, it is necessary to provide a navigation method, system, computer device, computer-readable storage medium, and computer program product that can perform modeling during operation, without preoperative CT, and with real-time registration without the need for additional markers, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a navigation method, the method comprising:

[0007] A target medical model is obtained, which is reconstructed based on an initial image scanned by a scanning device;

[0008] Based on the target medical model, operational planning information is generated;

[0009] The real-time image obtained by the scanning device is acquired, and the real-time image is registered with the target medical model to obtain a first registration relationship;

[0010] Obtain the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image;

[0011] The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the operation planning information.

[0012] In one embodiment, after acquiring the real-time image scanned by the scanning device and registering the real-time image with the target medical model to obtain a first registration relationship, the method further includes:

[0013] Based on the location of the target in the real-time image, update the location information of the target medical model;

[0014] The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes:

[0015] The operation planning information is updated based on the updated location information of the target medical model;

[0016] The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the updated operation planning information.

[0017] In one embodiment, obtaining the target medical model includes:

[0018] Acquire the current initial image scanned by the scanning device;

[0019] Target information is extracted from the current initial image, and a 3D point cloud of the current target is generated based on the extracted target information;

[0020] Register the current target 3D point cloud with the previous frame initial image;

[0021] Update the voxel values ​​of the corresponding 3D point cloud based on the registration results;

[0022] The target medical model is obtained by reconstructing the target using the voxel values ​​of the updated 3D point cloud.

[0023] In one embodiment, the extraction of target information from the current initial image includes:

[0024] Calculate the histogram of the current initial image, and extract the target region based on the histogram;

[0025] Based on the depth value of the target area and the coordinates of the voxel in the camera coordinate system, the distance from the voxel to the object surface is calculated.

[0026] Target information is generated based on the distance from the voxel to the object surface.

[0027] In one embodiment, updating the voxel values ​​of the corresponding 3D point cloud based on the registration result includes:

[0028] Obtain the voxel values ​​and weights of the current target 3D point cloud and the corresponding voxels of the previous frame initial image;

[0029] Based on the voxel values ​​of the current target 3D point cloud and the corresponding voxels of the previous frame initial image, and the corresponding voxel values ​​are updated with weights, the voxel values ​​of the current target 3D point cloud and the previous frame initial image that are not registered are retained.

[0030] In one embodiment, before obtaining the voxel values ​​and weights of the current target 3D point cloud and the corresponding voxels of the previous frame initial image, the method further includes:

[0031] The weights of the current target 3D point cloud are calculated based on the weights of the voxels corresponding to the initial image of the previous frame and the target weights.

[0032] In one embodiment, after generating the operation planning information based on the target medical model, the method further includes:

[0033] The operation boundary is generated based on the operation planning information;

[0034] The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes:

[0035] Based on the first coordinate relationship, the first registration relationship, and the operation planning information, the real-time position of the execution tool is determined;

[0036] When the real-time position exceeds the operation boundary, the system controls the execution of the task to stop and / or issues an alarm.

[0037] In one embodiment, the first coordinate relationship between the coordinate system corresponding to the acquisition execution tool and the image coordinate system of the real-time image includes at least one of the following:

[0038] Based on the position of the execution tool in the robotic arm coordinate system, the initial coordinates of the robotic arm base, the robotic arm's own posture, the robotic arm end-effector coordinate system, the rigid connection posture between the robotic arm end-effector and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image; or

[0039] Based on the handheld device end coordinate system, the rigid connection posture between the handheld device and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image.

[0040] In one embodiment, the scanning device includes at least two, wherein one portion of the scanning device is used to scan the operating area, and the other portion is used to scan the environmental area; the method further includes:

[0041] Environmental information is generated based on the environmental image scanned by the scanning device in the scanned environmental area;

[0042] Obtain the second coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the environment image;

[0043] The execution tool performs obstacle avoidance navigation based on the second coordinate relationship and the environmental information.

[0044] In one embodiment, the step of navigating the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes:

[0045] Based on the first coordinate relationship, the first registration relationship, and the operation planning information, navigation information for the execution tool is generated and displayed.

[0046] Secondly, this application also provides a navigation system, the system comprising:

[0047] A scanning device used to scan and obtain initial and real-time images;

[0048] Execution tools are used to perform operations according to the operation plan information;

[0049] The controller is configured to execute the navigation method described above based on the initial image and the real-time image obtained by the scanning device, so as to navigate the execution tool.

[0050] In one embodiment, the scanning device is mounted on a robotic arm; or the scanning device is mounted on a handheld device.

[0051] In one embodiment, the scanning device includes at least two, one of which is used to scan the operating area, and the other is used to scan the environmental area.

[0052] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0053] Fourthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0054] The aforementioned navigation method, system, computer equipment, storage medium, and computer program product reconstruct the target medical image based on the initial image scanned by the scanning device. This eliminates the need for preoperative CT scans of the surgical limb, reducing radiation hazards and shortening surgical preparation time. Furthermore, during operation, real-time images scanned by the scanning device are acquired, and the real-time images are registered with the target medical model to obtain a first registration relationship. Based on image registration, there is no need to add positioning markers to the target area, avoiding additional trauma. Finally, based on the first coordinate relationship, the first registration relationship, and operation planning information, the execution tool is navigated. This ensures that the scanning device is aligned with the surgical area and is less likely to be obstructed, resulting in more accurate navigation. Attached Figure Description

[0055] Figure 1 Here is a system block diagram of a navigation system in one embodiment;

[0056] Figure 2 This is a schematic diagram of a navigation system when the scanning device is mounted on a robotic arm in one embodiment.

[0057] Figure 3 This is a schematic diagram of the first connection between the scanning device and the robotic arm in one embodiment;

[0058] Figure 4 This is a schematic diagram of a single scanning device in one embodiment;

[0059] Figure 5 This is a schematic diagram of a multi-scanning device in one embodiment;

[0060] Figure 6 This is a schematic diagram of a navigation system in one embodiment where the scanning device is installed on a handheld device;

[0061] Figure 7 This is a schematic diagram of the second connection between the scanning device and the robotic arm in one embodiment;

[0062] Figure 8 This is a schematic diagram of navigation operations in one embodiment;

[0063] Figure 9 This is a schematic diagram of an embodiment where the execution tool is a oscillating saw;

[0064] Figure 10 This is a schematic diagram of a drill being used as the execution tool in one embodiment;

[0065] Figure 11 This is a schematic diagram of an embodiment where the execution tool is an osteotomy plate;

[0066] Figure 12 This is a flowchart illustrating a navigation method in one embodiment;

[0067] Figure 13 This is a flowchart illustrating the steps involved in generating a target medical model in one embodiment.

[0068] Figure 14 This is a rough flowchart illustrating the security control process in one embodiment;

[0069] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] The navigation method provided in this application embodiment can be applied to, for example, Figure 1 The navigation controller shown includes a scanning device 100, an execution tool 200, and a controller 300. The scanning device 100 is used to scan and obtain initial and real-time images; the execution tool 200 is used to operate according to the operation plan information; and the controller 300 is used to navigate the execution tool 200 based on the initial and real-time images obtained by the scanning device 100. The scanning device 100 may optionally be a depth camera, thus enabling it to scan the operation area, such as the patient's surgical area, thereby obtaining initial and real-time images. This eliminates the need for preoperative CT scans, reducing patient radiation exposure and shortening preoperative preparation time, making it suitable for emergency surgeries. No additional positioning markers are required for the affected limb, and no additional trauma is caused outside the surgical area.

[0072] Specifically, the scanning device 100 can be installed in at least two ways, one of which is that the scanning device 100 is mounted on the robotic arm 400, specifically in conjunction with... Figure 2 As shown, the scanning device 100 is mounted on the robotic arm 400, which is placed beside the operating table to ensure that the scanning device is directly facing the patient's operated limb 500. This is combined with... Figure 3 As shown, the scanning device 100 and the robotic arm 400 are connected and fixed through a quick-release structure 600. Each hole 601 is a positioning position for a pin. The scanning device 100 can be rotated to select different positioning positions, and each positioning position has a specific number. After the center point of the end tool of the robotic arm 400 is determined through this positioning position, the conversion relationship between the center of the end tool and the scanning device 100 is obtained. That is, based on the initial coordinates of the robotic arm base, the real-time coordinates of the real-time image are obtained based on the initial coordinates (robotic arm base coordinate system).

[0073] In practical applications, the scanning device 100 can scan and acquire initial images of the patient's operated limb, reconstruct a target medical model containing pose information, such as a skeletal model, and perform surgical planning based on the reconstructed skeletal model. During the surgery, the scanning device 100 scans and acquires real-time images of the patient's operated limb, registers the real-time images with the target medical model to obtain a first registration relationship. For example, given the initial coordinates RTBase of the robotic arm base, the coordinate system RTTool of the robotic arm end effector, the coordinate system RTCam of the real-time image, the posture of the robotic arm itself RTRobot, and the posture RTTtoC of the rigid connection between the end effector of the robotic arm 400 and the scanning device 100. Based on the initial coordinates of the robotic arm base, the real-time coordinates of the real-time image based on the initial coordinates (robotic arm base coordinate system) are obtained as: RTBase*RTRobot*RTTool*RTTtoC*RTCam. Thus, given the position of the execution tool in the robotic arm base coordinate system, the pose of the execution tool 200 in the planned real-time coordinate system is obtained, thereby guiding the execution tool 200 to the planned surgical location to complete the surgery.

[0074] In one optional embodiment, the scanning device 100 includes one unit, which is used to scan the operating area, specifically in conjunction with... Figure 4 As shown, the scanning device 100 is mounted on the robotic arm 400. After the patient's affected area is exposed during the operation, the scanning device 100 is placed so that the affected area is within the field of view of the scanning device 100. The field of view of the scanning device 100 is a fixed area, and it is necessary to ensure that the affected area is within the field of view during the operation.

[0075] In another alternative embodiment, the scanning device 100 includes at least two, wherein one scanning device 100 is used to scan the operating area, and the other scanning device 100 is used to scan the environmental area, specifically in combination with... Figure 5 As shown, the scanning device 100 is installed on the robotic arm 400. There may be multiple scanning devices 100 to collect environmental data in real time and establish a complete information on the surrounding environment. When the robotic arm 400 actively moves to the planned target position, if there is an obstacle in the trajectory, the controller will control the robotic arm 400 to stop moving before the collision because the scanning device 100 collects and judges it as an obstacle, thus realizing a safe and active collision avoidance model.

[0076] Another installation method involves mounting the scanning device 100 onto the handheld device 700, such as... Figure 6 As shown, the scanning device 100 is mounted on the handheld device 700, and the doctor holds it to the affected area, ensuring that the affected area is within the field of view of the scanning device 100. Combined with... Figure 7As shown, the scanning device 100 is connected to the handheld device 700, which includes a wireless module 701 and a power module 702. The scanning device 100 acquires the real-time coordinates of the skeleton under the scanning device and transmits them to the controller 300 via the wireless module 701. Figure 8 The controller 300 processes the real-time coordinates obtained from the scan using software, rendering and displaying a three-dimensional model of the patient's skeleton. The doctor observes the surgical plan displayed on the image and operates the handheld device 700, placing it at the planned surgical position to perform the surgery. Specifically, if an unexpected osteotomy occurs during the process guided by the interface image 800 to the planned real-time coordinates, the controller 300 sends a power-off signal, cutting off power to the handheld device 700's power module 702, preventing further osteotomy and ensuring precise and safe boundary protection.

[0077] In practical applications, the scanning device 100 scans and acquires data of the patient's operated limb to obtain an initial image, reconstructs a target medical model containing pose information, such as a skeletal model, and performs surgical planning based on the reconstructed skeletal model. During the surgery, the scanning device 100 scans and acquires real-time images of the patient's operated limb, registers the real-time images and the target medical model to obtain a first registration relationship. For example, given the real-time image coordinate system RTCam, the handheld device end coordinate system RTTool, and the rigid connection posture RTTtoC between the handheld device 700 and the scanning device 100, the real-time coordinates of the execution tool on the handheld device in the scanning device coordinate system can be obtained: RTTool*RTTtoC*RTCam. Based on the real-time coordinates in the scanning device, the interface image displays the surgical planning results and the real-time pose of the execution tool 200 in the image. The doctor guides the execution tool 200 to move to the planned real-time coordinates according to the interface image. In this embodiment, the use of a handheld device retains the doctor's traditional surgical experience while providing precise navigation and safety protection to prevent accidental bone resection.

[0078] In one embodiment, the execution tool is at least one of a oscillating saw, a grinding drill, and a bone cutting plate. Figures 9 to 11 As shown, where Figure 9 The execution tool in this case is a oscillating saw. Figure 10 The tool used for execution is a grinding drill. Figure 11 The execution tool used is an osteotomy plate.

[0079] This embodiment can be applied to knee replacement surgery, eliminating the need for preoperative CT scans. During the surgery, the robotic arm's end effector connects to tools such as a oscillating saw, bone drill, or osteotomy guide plate. Based on the initial coordinates of the robotic arm's base, real-time images are obtained using the real-time coordinates of the tools within the initial coordinates (robotic arm base coordinate system). Knowing the position of the tools within the robotic arm base coordinate system, the tool's pose in the planned real-time coordinate system is obtained. This guides the tools to the planned area, completing the surgical plan. During the surgery, if the tools move beyond the planned boundaries, the controller triggers protection, cutting off power to the tools or locking the robotic arm to prevent unintended osteotomy. Furthermore, no positioning target installation is required during the surgery, avoiding damage to the patient's tissues.

[0080] In one embodiment, such as Figure 12 As shown, a navigation method is provided, which is applied to Figure 1 Taking the controller in the example, the following steps are included:

[0081] S1202: Obtain the target medical model, which is reconstructed based on the initial image scanned by the scanning device.

[0082] Specifically, the target medical model is reconstructed based on initial images scanned by a scanning device. Before the procedure, such as before surgery, the scanning device acquires complete information about the entire operating area, such as the surgical area. This is achieved by acquiring multiple initial images through multiple scans, and the target medical model is reconstructed based on the acquired complete information. In practical applications, during the initial model acquisition during surgery, the scanning device is used to acquire complete skeletal information of the surgical area, and a three-dimensional model is reconstructed based on this complete skeletal information to obtain the target medical model.

[0083] Optionally, the generation of the target medical model may include: performing multiple consecutive scans using a scanning device to acquire multiple initial images, identifying the target in the multiple initial images, subsequently registering the multiple initial images based on the target, and finally reconstructing the target medical model based on the registration results.

[0084] It should be noted that since the target medical model is relative to the patient, as long as the patient remains unchanged, the target medical model will not change. Only the position of the target medical model may change, such as when the patient moves and the position of a specific part of the target medical model changes. However, these changes can be captured by real-time images subsequently acquired by the scanning device and updated to the target medical model through real-time registration.

[0085] The scanning device is positioned directly facing the operating area, making it less susceptible to obstruction and thus ensuring navigation accuracy.

[0086] S1204: Generate operational planning information based on the target medical model.

[0087] Specifically, the operation planning information refers to the surgical planning information, which is used by doctors to plan surgery for the affected area. It is not limited to planning the screw placement path, prosthesis planning, and other surgical planning methods. In this embodiment, there are no specific restrictions on the operation planning information. Any operation information set by the doctor based on the target medical model can be used as operation planning information. Moreover, the operation planning information is generated based on the target medical model. Therefore, the operation planning information and the target medical model have a positional relationship. That is to say, while the doctor sets the operation planning information on the target medical model, the controller generates the positional relationship between the target medical model and the operation planning information.

[0088] S1206: Acquire the real-time image obtained by the scanning device and register the real-time image with the target medical model to obtain the first registration relationship.

[0089] Specifically, the real-time image is obtained by the scanning device during operation. The real-time image can refer to each frame scanned by the scanning device, or it can be a periodically scanned image, such as randomly sampling a scanned image every second as a real-time image. No specific limitation is made here.

[0090] The registration of real-time images with the target medical model can be target-based, such as registration based on skeletal features. In practical applications, the controller acquires real-time images and extracts skeletal information from them. Since the scanning device is a depth camera, the real-time images are depth images, containing depth information for each point. Therefore, skeletal information can be extracted based on the depth information, and the extracted skeletal information is then registered with the target medical model. Registration based on skeletal features can determine the initial registration relationship between the real-time images and the target medical model.

[0091] S1208: Obtain the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image.

[0092] Specifically, the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image is related to the installation method of the scanning device. Different installation methods correspond to different calculation methods of the first coordinate relationship, which can be found below.

[0093] S1210: Navigate the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information.

[0094] Based on the first coordinate relationship, the first registration relationship, and the operation planning information, the position of the execution tool and the operation planning information can be mapped to the target medical model or the position of the execution tool can be mapped to the real-time image, thereby enabling navigation of the execution tool based on the operation planning information.

[0095] For simplicity, the example given is mapping all information to a real-time image. Based on a first coordinate relationship, the position of the execution tool can be mapped to the real-time image. Based on a first registration relationship, the target medical model can be mapped to the real-time image. Furthermore, since the operation planning information and the target medical model have a positional relationship, the operation planning information can also be mapped to the real-time image, thus enabling navigation of the execution tool based on the real-time image. Optionally, the operation planning information and the position of the execution tool can be displayed simultaneously in the real-time image, thereby enabling navigation of the execution tool. In other embodiments, all information can also be mapped to the target medical model. This is not specifically limited here; it is only necessary to note that the mapped target medical model is the target medical model whose position is updated based on the real-time image.

[0096] The aforementioned navigation method reconstructs the target medical image based on the initial image scanned by the scanning device. This eliminates the need for preoperative CT scans of the surgical limb, reducing radiation hazards and shortening surgical preparation time. Furthermore, during the operation, real-time images scanned by the scanning device are acquired, and these real-time images are registered with the target medical model to obtain a first registration relationship. Based on image registration, there is no need to add positioning markers to the target area, avoiding additional trauma. Finally, based on the first coordinate relationship, the first registration relationship, and the operation planning information, the execution tool is navigated. This ensures that the scanning device is aligned with the surgical area and is less likely to be obstructed, resulting in more accurate navigation.

[0097] In one embodiment, after acquiring a real-time image scanned by a scanning device and registering the real-time image with a target medical model to obtain a first registration relationship, the method further includes: updating the position information of the target medical model based on the position of the target in the real-time image; and navigating the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information, including: updating the operation planning information based on the updated position information of the target medical model; and navigating the execution tool based on the first coordinate relationship, the first registration relationship, and the updated operation planning information.

[0098] In practical applications, it is best to keep the patient still so that the position of the target medical model remains unchanged. However, the patient may move during the operation, necessitating an update to the position of the target medical model. Therefore, feature registration is used when registering the real-time image with the target medical model. This involves registering the skeleton in the real-time image with the features of the target medical model to obtain the registered features. The position of the corresponding registered feature in the real-time image, or the correspondence between that position and the position of the corresponding registered feature in the target medical model, is then obtained. Based on this correspondence, the position of the entire target medical model is updated.

[0099] Since the positional relationship between the target medical model and the operation planning information is fixed, the position of the operation planning information can be updated after the position of the target medical model is updated. Consequently, the operation planning information mapped to the real-time image can also be updated, thereby navigating the execution tool based on the updated operation planning information.

[0100] In the above embodiments, intraoperative real-time registration is performed, and the registration accuracy is not affected by bone movement.

[0101] In one embodiment, see Figure 13 As shown, the process of generating the target medical model includes:

[0102] S1302: Acquire the current initial image scanned by the scanning device.

[0103] Specifically, when the target medical model is generated, the scanning device acquires multiple initial images, and the initial image obtained from the current scan is used as the current initial image.

[0104] S1304: Extract target information from the current initial image and generate a 3D point cloud of the current target based on the extracted target information.

[0105] Specifically, target information refers to the information of the target in the current initial image, such as the skeletal information in the current initial image, that is, extracting skeletal information from the current initial image.

[0106] In one optional embodiment, target information extraction is performed on the current initial image, including: calculating the histogram of the current initial image and extracting the target region based on the histogram; calculating the distance from the voxel to the object surface based on the depth value of the target region and the coordinates of the voxel in the camera coordinate system; and generating target information based on the distance from the voxel to the object surface.

[0107] Since the scanning device is a depth camera, the acquired initial image is a depth image, containing depth information, or depth values, for each point. Extracting target information from this initial image involves segmenting the skeleton. To do this, the histogram of the initial image is first calculated, and the maximum peak value D of the depth values ​​within a given threshold range is found. max If a deviation Δ is set, then the depth map of the skeleton is...

[0108]

[0109] Among them, D i (u, v) is the depth value of the voxel (x, y, z) back-projected onto the current frame's depth image, D b (u, v) represents the depth map of the skeleton, D s (u, v) represents the current initial image.

[0110] The truncated signed distance function (TSDF) value of a voxel represents the truncated weighted directed distance of each voxel to the nearest object surface along the sensor's line of sight. A positive TSDF value indicates that the point is not occluded by a surface, while a negative TSDF value indicates that the voxel is located inside a surface. The formula for calculating the voxel's directed distance SDF is as follows:

[0111] d i (x, y, z) = D i (u, v)-z i

[0112] Where d i (x, y, z) represents the distance from voxel (x, y, z) to the object surface in the current frame, D i (u, v) is the depth value of the voxel (x, y, z) back-projected onto the current frame's depth image, where z i This represents the z-axis coordinate of the voxel in the camera coordinate system. The SDF value is then truncated to obtain the TSDF value.

[0113]

[0114] Where d′ i (x, y, z) represents the truncated TSDF value, and maxtruncation and mintruncation represent the truncation range, thereby extracting the target information in the current initial image.

[0115] Based on the extracted target information, a 3D point cloud of the current target is generated, and the truncated TSDF value is used as the voxel value at the corresponding position.

[0116] One point to note is that before registering the real-time images acquired in subsequent operations with the target medical model, the real-time images can be identified to obtain the target. The process of real-time image identification can be found in the target information generation process described above.

[0117] S1306: Register the current target 3D point cloud with the previous frame's initial image.

[0118] S1308: Update the voxel values ​​of the corresponding 3D point cloud based on the registration results.

[0119] Specifically, the process of registering the current target 3D point cloud with the previous frame initial image can be done through ICP registration. In other embodiments, registration can also be done through other point cloud registration methods to obtain the registered part of the current initial image and the unregistered part of the previous frame initial image.

[0120] In one optional embodiment, updating the voxel values ​​of the corresponding 3D point cloud based on the registration result includes: obtaining the voxel values ​​and weights of the voxels corresponding to the current target 3D point cloud and the previous frame initial image; updating the corresponding voxel values ​​based on the voxel values ​​and weights of the voxels corresponding to the current target 3D point cloud and the previous frame initial image, and retaining the voxel values ​​of the voxels of the current target 3D point cloud that are not registered with the previous frame initial image.

[0121] For the unregistered parts, the voxel values ​​of the unregistered voxels in the current target 3D point cloud and the previous initial image are directly retained. For the registered parts, the voxel values ​​are updated based on the voxel values ​​of the corresponding voxels in the current target 3D point cloud and the previous initial image, as well as the weights. For example, the voxel values ​​are updated based on the voxel values ​​of the corresponding voxels in the current target 3D point cloud and the previous initial image, as well as the weights.

[0122] The weight of the current target 3D point cloud is calculated based on the weight of the voxel corresponding to the initial image of the previous frame and the target weight. Specifically, the target weight is the maximum weight, and generally, the maximum weight is 1, which can remove the influence of dynamic objects in the scene.

[0123]

[0124] Among them, W i (x, y, z) = min (max weight, W i-1 (x, y, z) + 1)

[0125] Among them, D i-1 (x, y, z) represents the distance from this voxel to the object's surface in the previous frame, W i-1 (x, y, z) represents the weight of this voxel in the previous frame, W i(x, y, z) represents the weight of the voxel in the current frame, and maxweight is the maximum weight. For ease of weight calculation, an initial weight is set, which is the weight of the initial image in the first frame. Generally, this initial weight is 1. A simple averaging of the TSDF yields good results.

[0126] S1310: Reconstruct the target medical model by performing target reconstruction on the voxel values ​​of the updated 3D point cloud.

[0127] Specifically, after all the initial image processing is completed, the voxel values ​​in the 3D point cloud are also updated. Finally, the target is reconstructed based on the updated voxel values ​​of the 3D point cloud to obtain the target medical model. For example, points with a TSDF value of 0 are obtained through interpolation calculation, and the 3D mesh is reconstructed to obtain the target medical model, i.e., the skeleton model.

[0128] In one embodiment, after generating operation planning information based on the target medical model, the method further includes: generating operation boundaries based on the operation planning information; navigating the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information, including: determining the real-time position of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information; and controlling the execution to stop moving when the real-time position exceeds the operation boundary, and / or issuing an alarm.

[0129] Specifically, the operational boundary is generated based on operational planning information, such as determining the operational route of the execution tool and determining the boundary of the operational route based on skeletal information, which is the operational boundary. The operational boundary can be understood as the boundary of the maximum safe area, such as the maximum safe area to avoid harming the patient.

[0130] During operation, the real-time position of the execution tool is determined based on the first coordinate relationship, the first registration relationship, and the operation planning information. For example, the position of the execution tool is mapped to the real-time image, and the operation boundary is mapped to the real-time image. In this way, it is determined whether the real-time position of the execution tool exceeds the operation boundary in the real-time image. If it does, the execution work is stopped and / or an alarm is triggered.

[0131] When the scanning device is installed on the robotic arm, if the execution tool moves beyond the planned boundary during the operation, the controller triggers protection, cutting off the power to the execution tool or locking the robotic arm to prevent unexpected bone resection.

[0132] When the scanning device is installed on the execution tool, if an unexpected osteotomy occurs during the operation, that is, when the execution tool moves beyond the planned boundary, the controller will send a power cut-off signal, the power to the handheld device will be cut off, and the osteotomy cannot be performed, thus ensuring precise and safe boundary protection.

[0133] In the above embodiments, when a doctor is performing surgery using the equipment, if the navigation exceeds the planned boundary, if a robotic arm is used, the robotic arm will brake and lock, and an alarm sound will be emitted to prevent the doctor from continuing the surgery beyond the expected scope; if a handheld device is used, the controller will cut off the power to the handheld device and emit an alarm sound to ensure safety.

[0134] In one embodiment, obtaining the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image includes at least one of the following: obtaining the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image based on the position of the execution tool in the robotic arm coordinate system, the initial coordinates of the robotic arm base, the posture of the robotic arm itself, the coordinate system of the robotic arm end effector, the rigid connection posture between the robotic arm end effector and the scanning device, and the image coordinate system of the real-time image; or obtaining the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image based on the coordinate system of the handheld device end effector, the rigid connection posture between the handheld device and the scanning device, and the image coordinate system of the real-time image.

[0135] Specifically, this embodiment mainly introduces the installation method of the scanning device. One method is to install the scanning device on the robotic arm. The scanning device is installed at the end of the robotic arm. Through a rigid connection, the position of the robotic arm end-effector tool in the coordinate system of the scanning device can be obtained. The specific position that the tool needs to reach for the surgical planning path can be obtained. Precise positioning can be achieved through robotic arm navigation.

[0136] Specifically, given the initial coordinates of the robotic arm base RTBase, the coordinate system of the robotic arm end effector RTTool, the real-time image coordinate system RTCam, the robotic arm's own posture RTRobot, and the rigid connection posture between the robotic arm end effector and the scanning device RTTtoC, the real-time coordinates of the real-time image based on the initial coordinates of the robotic arm base are obtained as: RTBase*RTRobot*RTTool*RTTtoC*RTCam.

[0137] In another embodiment, the scanning device is mounted on a handheld device, for example, on the end of the handheld device, as can be seen in [reference needed]. Figure 6 This allows surgeons to maintain their surgical habits, using traditional instruments in conjunction with a navigation and positioning controller. During surgery, surgeons can verify the accuracy of the surgical outcome in real time according to the surgical plan, avoiding unforeseen complications. The scanning device is positioned close to the patient's area for convenient intraoperative observation of the surgical site, preventing obstruction of the camera's view.

[0138] Specifically, given the real-time image coordinate system RTCam, the handheld device end coordinate system RTTool, and the rigid connection posture between the handheld device and the scanning device RTTtoC, the real-time coordinates of the execution tool on the handheld device in the scanning device coordinate system can be obtained as: RTTool*RTTtoC*RTCam.

[0139] The above embodiments provide a method for calculating the first coordinate relationship.

[0140] In one embodiment, the scanning device includes at least two devices, one of which is used to scan the operating area and the other is used to scan the environmental area. The method further includes: generating environmental information based on the environmental image scanned by the scanning device of the scanning area; obtaining a second coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the environmental image; and performing obstacle avoidance navigation on the execution tool based on the second coordinate relationship and the environmental information.

[0141] Continue to combine Figure 5 As shown, the scanning device is installed on the robotic arm. There may be multiple scanning devices to collect environmental data in real time and establish a complete information on the surrounding environment. When the robotic arm actively moves to the planned target position, if there is an obstacle in the trajectory, the controller will control the robotic arm to stop moving before the collision because the scanning device collects and judges it as an obstacle, thus realizing a safe and active collision avoidance model.

[0142] In one embodiment, navigating the execution tool based on the first coordinate relationship, the first registration relationship, and operation planning information includes: generating and displaying navigation information for the execution tool based on the first coordinate relationship, the first registration relationship, and operation planning information.

[0143] Specifically, in combination Figure 8 As shown, the scanning device connects to a handheld device. During the procedure, the scanning device connects to a computer via a wireless communication module, transmitting the acquired images back to the computer. Software processes and renders a 3D model of the patient's skeleton. The surgeon observes the surgical plan displayed on the image, operates the handheld device, and positions it at the planned surgical location to perform the surgery. This approach retains the surgeon's traditional surgical habits while providing precise surgical planning and navigation.

[0144] To ensure that those skilled in the art fully understand the security controls of this application, please refer to the following details. Figure 14As shown, different installation methods of the scanning device correspond to different safety control strategies. When the scanning device is installed on the robotic arm, it generates operating boundaries based on the operation planning information to perform tool-guided surgery. If the planned boundaries are exceeded, the robotic arm brake locks, and the controller issues an alarm. On the other hand, the scanning device collects environmental information. When the robotic arm moves autonomously, if the scanning device observes a human body entering the robotic arm's movement trajectory, it promptly sends a robotic arm stop signal, and the robotic arm brake locks.

[0145] When the scanning device is mounted on the handheld device, the position of the execution tool is known in real-time coordinates. The surgery is performed under the guidance of the execution tool. If the operation exceeds the planned boundary, the controller cuts off the power supply to the handheld device.

[0146] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0147] Based on the same inventive concept, this application also provides a navigation device for implementing the navigation method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more navigation device embodiments provided below can be found in the limitations of the navigation method described above, and will not be repeated here.

[0148] In one embodiment, a navigation device is provided, comprising: a model acquisition module, a planning information generation module, a first registration module, a first coordinate relationship acquisition module, and a navigation module, wherein:

[0149] The model acquisition module is used to acquire a target medical model, which is reconstructed based on an initial image scanned by a scanning device.

[0150] The planning information generation module is used to generate operational planning information based on the target medical model.

[0151] The first registration module is used to acquire the real-time image scanned by the scanning device and register the real-time image with the target medical model to obtain a first registration relationship;

[0152] The first coordinate relationship acquisition module is used to acquire the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image;

[0153] The navigation module is used to navigate the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information.

[0154] In one embodiment, the navigation device further includes:

[0155] The location update module is used to update the location information of the target medical model based on the location of the target in the real-time image;

[0156] The navigation module is also used to update the operation planning information based on the updated location information of the target medical model; and to navigate the execution tool based on the first coordinate relationship, the first registration relationship, and the updated operation planning information.

[0157] In one embodiment, the model acquisition module is further configured to acquire the current initial image scanned by the scanning device; extract target information from the current initial image and generate a current target 3D point cloud based on the extracted target information; register the current target 3D point cloud with the previous frame initial image; update the voxel values ​​of the corresponding 3D point cloud based on the registration result; and reconstruct the target based on the updated voxel values ​​of the 3D point cloud to obtain the target medical model.

[0158] In one embodiment, the model acquisition module is further configured to calculate the histogram of the current initial image and extract the target region based on the histogram; calculate the distance from the voxel to the object surface based on the depth value of the target region and the coordinates of the voxel in the camera coordinate system; and generate target information based on the distance from the voxel to the object surface.

[0159] In one embodiment, the model acquisition module is further configured to acquire the voxel values ​​and weights of the corresponding voxels of the current target 3D point cloud and the previous frame initial image; update the corresponding voxel values ​​based on the voxel values ​​of the current target 3D point cloud and the previous frame initial image, and the weights, and retain the voxel values ​​of the voxels of the current target 3D point cloud that are not registered with the previous frame initial image.

[0160] In one embodiment, the model acquisition module is further configured to calculate the weight of the current target 3D point cloud based on the weight of the voxel corresponding to the previous frame initial image and the target weight.

[0161] In one embodiment, the navigation device further includes:

[0162] An operation boundary generation module is used to generate operation boundaries based on the operation planning information;

[0163] The navigation module is also used to determine the real-time position of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information; when the real-time position exceeds the operation boundary, control the execution tool to stop moving, and / or issue an alarm.

[0164] In one embodiment, the first coordinate relationship acquisition module obtains the first coordinate relationship in at least one of the following ways: based on the position of the execution tool in the robotic arm coordinate system, the initial coordinates of the robotic arm base, the posture of the robotic arm itself, the coordinate system of the robotic arm end effector, the rigid connection posture between the robotic arm end effector and the scanning device, and the image coordinate system of the real-time image, the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image is obtained; or based on the coordinate system of the handheld device end effector, the rigid connection posture between the handheld device and the scanning device, and the image coordinate system of the real-time image, the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image is obtained.

[0165] In one embodiment, the scanning device includes at least two devices, one of which is used to scan an operating area, and the other is used to scan an environmental area; the device further includes:

[0166] The obstacle avoidance module is used to generate environmental information based on the environmental image scanned by the scanning device in the scanned environmental area; obtain a second coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the environmental image; and perform obstacle avoidance navigation for the execution tool based on the second coordinate relationship and the environmental information.

[0167] In one embodiment, the navigation module is further configured to generate and display navigation information of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information.

[0168] The modules in the aforementioned navigation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0169] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a controller bus, and the communication interface, display unit, and input device are connected to the controller bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operation controller and computer program. The internal memory provides an environment for the operation controller and computer program stored in the non-volatile storage media to run. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a navigation method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0170] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0172] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0174] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0175] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A navigation system, characterized in that, The system includes: A scanning device used to scan and obtain initial and real-time images; Execution tools are used to perform operations according to the operation plan information; The controller is used to perform the following steps: A target medical model is obtained, which is reconstructed based on an initial image scanned by a scanning device; Based on the target medical model, operational planning information is generated; The real-time image obtained by the scanning device is acquired, and the real-time image is registered with the target medical model to obtain a first registration relationship; Obtain the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image; The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the operation planning information; The acquisition of the target medical model includes: Acquire the current initial image scanned by the scanning device; Extracting target information from the current initial image includes: calculating the histogram of the current initial image and extracting the target region based on the histogram; calculating the distance from the voxel to the object surface based on the depth value of the target region and the coordinates of the voxel in the camera coordinate system; and generating target information based on the distance from the voxel to the object surface. Based on the extracted target information, a target medical model is obtained.

2. The system according to claim 1, characterized in that, After acquiring the real-time image scanned by the scanning device and registering the real-time image with the target medical model to obtain a first registration relationship, the method further includes: Based on the location of the target in the real-time image, update the location information of the target medical model; The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: The operation planning information is updated based on the updated location information of the target medical model; The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the updated operation planning information.

3. The system according to claim 1, characterized in that, The process of obtaining the target medical model based on the extracted target information includes: Generate a 3D point cloud of the current target based on the extracted target information; Register the current target 3D point cloud with the previous frame initial image; Update the voxel values ​​of the corresponding 3D point cloud based on the registration results; The target medical model is obtained by reconstructing the target using the voxel values ​​of the updated 3D point cloud.

4. The system according to claim 3, characterized in that, The process of updating the voxel values ​​of the corresponding 3D point cloud based on the registration results includes: Obtain the voxel values ​​and weights of the current target 3D point cloud and the corresponding voxels of the previous frame initial image; Based on the voxel values ​​of the current target 3D point cloud and the corresponding voxels of the previous frame initial image, and the corresponding voxel values ​​are updated with weights, the voxel values ​​of the current target 3D point cloud and the previous frame initial image that are not registered are retained.

5. The system according to claim 4, characterized in that, Before obtaining the voxel values ​​and weights of the corresponding voxels of the current target 3D point cloud and the previous frame initial image, the method further includes: The weights of the current target 3D point cloud are calculated based on the weights of the voxels corresponding to the initial image of the previous frame and the target weights.

6. The system according to any one of claims 1 to 5, characterized in that, After generating the operation planning information based on the target medical model, the process further includes: The operation boundary is generated based on the operation planning information; The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: Based on the first coordinate relationship, the first registration relationship, and the operation planning information, the real-time position of the execution tool is determined; When the real-time position exceeds the operating boundary, the execution tool is controlled to stop moving, and / or an alarm is triggered.

7. The system according to any one of claims 1 to 5, characterized in that, The first coordinate relationship between the coordinate system corresponding to the acquisition execution tool and the image coordinate system of the real-time image includes at least one of the following: Based on the position of the execution tool in the robot arm coordinate system, the initial coordinates of the robot arm base, the robot arm's own posture, the robot arm end coordinate system, the rigid connection posture between the robot arm end and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image. or Based on the handheld device end coordinate system, the rigid connection posture between the handheld device and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image.

8. The system according to claim 7, characterized in that, The scanning device includes at least two devices, one of which is used to scan the operating area, and the other is used to scan the environmental area; the controller is also used to perform the following steps: Environmental information is generated based on the environmental image scanned by the scanning device in the scanned environmental area; Obtain the second coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the environment image; The execution tool performs obstacle avoidance navigation based on the second coordinate relationship and the environmental information.

9. The system according to claim 1, characterized in that, The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: Based on the first coordinate relationship, the first registration relationship, and the operation planning information, navigation information for the execution tool is generated and displayed.

10. The system according to claim 1, characterized in that, The scanning device is mounted on a robotic arm; or, the scanning device is mounted on a handheld device.

11. The system according to claim 10, characterized in that, The scanning device includes at least two, one of which is used to scan the operating area, and the other is used to scan the environmental area.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the following steps: A target medical model is obtained, which is reconstructed based on an initial image scanned by a scanning device; Based on the target medical model, operational planning information is generated; The real-time image obtained by the scanning device is acquired, and the real-time image is registered with the target medical model to obtain a first registration relationship; Obtain the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image; The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the operation planning information; The acquisition of the target medical model includes: Acquire the current initial image scanned by the scanning device; Extracting target information from the current initial image includes: calculating the histogram of the current initial image and extracting the target region based on the histogram; calculating the distance from the voxel to the object surface based on the depth value of the target region and the coordinates of the voxel in the camera coordinate system; and generating target information based on the distance from the voxel to the object surface. Based on the extracted target information, a target medical model is obtained.

13. The computer-readable storage medium according to claim 12, characterized in that, After the computer program, when executed by the processor, acquires the real-time image scanned by the scanning device and registers the real-time image with the target medical model to obtain a first registration relationship, it further includes: Based on the location of the target in the real-time image, update the location information of the target medical model; The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: The operation planning information is updated based on the updated location information of the target medical model; The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the updated operation planning information.

14. The computer-readable storage medium according to claim 12, characterized in that, When the computer program is executed by the processor, it implements the extraction of target information to obtain a target medical model, including: Generate a 3D point cloud of the current target based on the extracted target information; Register the current target 3D point cloud with the previous frame initial image; Update the voxel values ​​of the corresponding 3D point cloud based on the registration results; The target medical model is obtained by reconstructing the target using the voxel values ​​of the updated 3D point cloud.

15. The computer-readable storage medium according to claim 14, characterized in that, The updating of the voxel values ​​of the corresponding 3D point cloud based on the registration results, implemented by the computer program when executed by the processor, includes: Obtain the voxel values ​​and weights of the current target 3D point cloud and the corresponding voxels of the previous frame initial image; Based on the voxel values ​​of the current target 3D point cloud and the corresponding voxels of the previous frame initial image, and the corresponding voxel values ​​are updated with weights, the voxel values ​​of the current target 3D point cloud and the previous frame initial image that are not registered are retained.

16. The computer-readable storage medium according to claim 15, characterized in that, Before the process of obtaining the voxel values ​​and weights of the corresponding voxels of the current target 3D point cloud and the previous frame initial image, the computer program, when executed by the processor, further includes: The weights of the current target 3D point cloud are calculated based on the weights of the voxels corresponding to the initial image of the previous frame and the target weights.

17. The computer-readable storage medium according to any one of claims 12 to 16, characterized in that, After the computer program is executed by the processor and generates the operation planning information based on the target medical model, it further includes: The operation boundary is generated based on the operation planning information; The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: Based on the first coordinate relationship, the first registration relationship, and the operation planning information, the real-time position of the execution tool is determined; When the real-time position exceeds the operating boundary, the execution tool is controlled to stop moving, and / or an alarm is triggered.

18. The computer-readable storage medium according to any one of claims 12 to 16, characterized in that, The first coordinate relationship between the coordinate system corresponding to the acquisition execution tool and the image coordinate system of the real-time image, implemented when the computer program is executed by the processor, includes at least one of the following: Based on the position of the execution tool in the robot arm coordinate system, the initial coordinates of the robot arm base, the robot arm's own posture, the robot arm end coordinate system, the rigid connection posture between the robot arm end and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image. or Based on the handheld device end coordinate system, the rigid connection posture between the handheld device and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image.

19. The computer-readable storage medium according to claim 18, characterized in that, The scanning device includes at least two devices, one of which is used to scan the operating area, and the other is used to scan the environmental area; when the computer program is executed by the processor, it further performs the following steps: Environmental information is generated based on the environmental image scanned by the scanning device in the scanned environmental area; Obtain the second coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the environment image; The execution tool performs obstacle avoidance navigation based on the second coordinate relationship and the environmental information.

20. The computer-readable storage medium according to claim 12, characterized in that, The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information, implemented by the computer program when executed by the processor, includes: Based on the first coordinate relationship, the first registration relationship, and the operation planning information, navigation information for the execution tool is generated and displayed.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it performs the following steps: acquiring a target medical model, which is reconstructed based on an initial image scanned by a scanning device; Based on the target medical model, operational planning information is generated; The real-time image obtained by the scanning device is acquired, and the real-time image is registered with the target medical model to obtain a first registration relationship; Obtain the first coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image; The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the operation planning information; The acquisition of the target medical model includes: Acquire the current initial image scanned by the scanning device; Extracting target information from the current initial image includes: calculating the histogram of the current initial image and extracting the target region based on the histogram; calculating the distance from the voxel to the object surface based on the depth value of the target region and the coordinates of the voxel in the camera coordinate system; and generating target information based on the distance from the voxel to the object surface. Based on the extracted target information, a target medical model is obtained.

22. The computer program product according to claim 21, characterized in that, After the computer program, when executed by the processor, acquires the real-time image scanned by the scanning device and registers the real-time image with the target medical model to obtain a first registration relationship, it further includes: Based on the location of the target in the real-time image, update the location information of the target medical model; The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: The operation planning information is updated based on the updated location information of the target medical model; The execution tool is navigated based on the first coordinate relationship, the first registration relationship, and the updated operation planning information.

23. The computer program product according to claim 21, characterized in that, When the computer program is executed by the processor, it implements the extraction of target information to obtain a target medical model, including: Generate a 3D point cloud of the current target based on the extracted target information; Register the current target 3D point cloud with the previous frame initial image; Update the voxel values ​​of the corresponding 3D point cloud based on the registration results; The target medical model is obtained by reconstructing the target using the voxel values ​​of the updated 3D point cloud.

24. The computer program product according to claim 23, characterized in that, The updating of the voxel values ​​of the corresponding 3D point cloud based on the registration result, implemented when the computer program is executed by the processor, includes: Obtain the voxel values ​​and weights of the current target 3D point cloud and the corresponding voxels of the previous frame initial image; Based on the voxel values ​​of the current target 3D point cloud and the corresponding voxels of the previous frame initial image, and the corresponding voxel values ​​are updated with weights, the voxel values ​​of the current target 3D point cloud and the previous frame initial image that are not registered are retained.

25. The computer program product according to claim 24, characterized in that, Before the process of obtaining the voxel values ​​and weights of the corresponding voxels of the current target 3D point cloud and the previous frame initial image when the computer program is executed by the processor, the process further includes: The weights of the current target 3D point cloud are calculated based on the weights of the voxels corresponding to the initial image of the previous frame and the target weights.

26. The computer program product according to any one of claims 21 to 25, characterized in that, After the computer program is executed by the processor and generates the operation planning information based on the target medical model, it further includes: The operation boundary is generated based on the operation planning information; The navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: Based on the first coordinate relationship, the first registration relationship, and the operation planning information, the real-time position of the execution tool is determined; When the real-time position exceeds the operating boundary, the execution tool is controlled to stop moving, and / or an alarm is triggered.

27. The computer program product according to any one of claims 21 to 25, characterized in that, The first coordinate relationship between the coordinate system corresponding to the acquisition execution tool and the image coordinate system of the real-time image, implemented when the computer program is executed by the processor, includes at least one of the following: Based on the position of the execution tool in the robot arm coordinate system, the initial coordinates of the robot arm base, the robot arm's own posture, the robot arm end coordinate system, the rigid connection posture between the robot arm end and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image. or Based on the handheld device end coordinate system, the rigid connection posture between the handheld device and the scanning device, and the image coordinate system of the real-time image, a first coordinate relationship is obtained between the coordinate system corresponding to the execution tool and the image coordinate system of the real-time image.

28. The computer program product according to claim 27, characterized in that, The scanning device includes at least two devices, one of which is used to scan the operating area, and the other is used to scan the environmental area; when the computer program is executed by the processor, it further performs the following steps: Environmental information is generated based on the environmental image scanned by the scanning device in the scanned environmental area; Obtain the second coordinate relationship between the coordinate system corresponding to the execution tool and the image coordinate system of the environment image; The execution tool performs obstacle avoidance navigation based on the second coordinate relationship and the environmental information.

29. The computer program product according to claim 21, characterized in that, When the computer program is executed by the processor, the navigation of the execution tool based on the first coordinate relationship, the first registration relationship, and the operation planning information includes: Based on the first coordinate relationship, the first registration relationship, and the operation planning information, navigation information for the execution tool is generated and displayed.

Images