An obstacle avoidance prompting method and device, medium and equipment

Abstract

The specification discloses an obstacle avoidance prompt method, device, medium and equipment. According to the pose of a human optical beacon collected by a tracker and the pose of a surgical equipment optical beacon, the actual position of the working end of the surgical equipment and the actual position of the human optical beacon are determined. A CT image containing the working end and the human optical beacon collected during surgery is obtained, and a nuclear magnetic image collected before surgery is obtained. The working end and the human optical beacon in the CT image are identified, and according to the actual position of the working end and the actual position of the human optical beacon, a first transpose matrix of the CT image and the tracker is determined. A second transpose matrix of the CT image and the nuclear magnetic image is determined, and according to the first transpose matrix and the second transpose matrix, the position of the working end in the nuclear magnetic image is determined. The target object in the nuclear magnetic image is identified, and the position of the target object in the nuclear magnetic image is determined. According to the position of the working end and the position of the target object, obstacle avoidance prompt information is determined and displayed.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an obstacle avoidance warning method, device, medium and equipment. Background Technology

[0002] Currently, endoscopic minimally invasive surgery is widely favored in clinical practice because it overcomes the drawbacks of traditional surgery, such as large incisions and long recovery periods. For example, minimally invasive spinal surgery, due to its smaller incisions, helps reduce the risk of postoperative pain and complications, and patients typically experience shorter recovery times. However, spinal endoscopic surgery can potentially damage nerve roots, cartilage tissue, and vertebrae, leading to problems such as limb numbness, radiating pain, muscle weakness, sensory abnormalities, and even paralysis.

[0003] In existing technologies, MRI images are typically acquired before surgery, and ultrasound images are acquired during surgery. The MRI images and ultrasound images are then registered to determine the position of surgical instruments in the lesion area and displayed to the doctor, thereby reducing the probability of surgical instrument injury to the patient during surgery.

[0004] However, ultrasound images contain far less information than MRI images, resulting in lower registration accuracy and less precise positioning of surgical instruments, still relying heavily on the surgeon's experience. Therefore, this manual provides an obstacle avoidance warning method, device, medium, and equipment. Summary of the Invention

[0005] This specification provides an obstacle avoidance warning method, device, medium, and equipment to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This manual provides a method for providing obstacle avoidance prompts, including:

[0008] Based on the pose of the human optical beacon and the pose of the surgical equipment optical beacon collected by the tracker, the actual position of the working end of the surgical equipment and the actual position of the human optical beacon are determined.

[0009] Acquire intraoperative CT images containing the working end and the human optical beacon, and acquire preoperative MRI images;

[0010] Identify the working end and the human optical beacon in the CT image, and determine the first transpose matrix of the CT image and the tracker based on the actual position of the working end and the actual position of the human optical beacon;

[0011] Determine the second transpose matrix of the CT image and the MRI image, and determine the position of the working end in the MRI image based on the first transpose matrix and the second transpose matrix;

[0012] Target object identification is performed on the nuclear magnetic resonance image to determine the location of the target object in the nuclear magnetic resonance image;

[0013] Based on the position of the working end and the position of the target object, obstacle avoidance prompts are determined and displayed.

[0014] Optionally, identifying the working end and the human optical beacon in the CT image, and determining the first transpose matrix of the CT image and the tracker based on the actual positions of the working end and the human optical beacon, specifically includes:

[0015] The working end and the human optical beacon in the CT image are determined using image recognition methods.

[0016] Based on the positions of the working end and the human optical beacon in the CT image, determine the first coordinates of the working end and the human optical beacon in the coordinate system of the CT image;

[0017] Based on the actual position of the working end and the actual position of the human optical beacon collected by the tracker, the second coordinates of the working end and the human optical beacon in the coordinate system of the tracker are determined;

[0018] Based on the first and second coordinates of the working end, and the first and second coordinates of the human optical beacon, determine the first transpose matrix of the coordinate system of the CT image and the coordinate system of the tracker.

[0019] Optionally, determining the second transpose matrix of the CT image and the MRI image specifically includes:

[0020] Determine the surgical region in the CT image and the surgical region in the MRI image;

[0021] The surgical region in the CT image and the surgical region in the MRI image are segmented to obtain a first segmented image of the surgical region in the CT image and a second segmented image of the surgical region in the MRI image.

[0022] Based on the first segmented image and the second segmented image, a second transpose matrix of the CT image and the MRI image is determined, wherein the first segmented image and the second segmented image are segmented images of the same part of the surgical area.

[0023] Optionally, the surgical area includes the spine, and the first segmented image and the second segmented image are two vertebrae at the same location of the spine;

[0024] Based on the first segmented image and the second segmented image, a second transpose matrix of the CT image and the MRI image is determined, specifically including:

[0025] Based on the positions of the first vertebra and the second vertebra in the first segmented image, determine the first coordinates of the first vertebra and the second vertebra in the first segmented image;

[0026] Based on the positions of the first and second vertebrae in the second segmented image, determine the third coordinates of the first and second vertebrae in the second segmented image;

[0027] Based on the first and third coordinates of the first vertebral body and the first and third coordinates of the second vertebral body, a second transpose matrix of the coordinate system of the CT image and the coordinate system of the MRI image is determined.

[0028] Optionally, the target object is a nerve root;

[0029] The process of identifying a target object in the MRI image and determining the location of the target object in the MRI image specifically includes:

[0030] Identify the surgical area in the MRI image;

[0031] The surgical area is segmented to determine the location of the nerve root within the surgical area.

[0032] Optionally, obstacle avoidance prompts are determined and displayed based on the position of the working end and the position of the target object, specifically including:

[0033] Determine the distance between the position of the working end and the position of the target object in the NMR image;

[0034] When the distance is less than a preset threshold, an obstacle avoidance prompt is generated and displayed, which is used to prompt the user to avoid the target object.

[0035] Optionally, the method further includes:

[0036] Determine the distance between the position of the working end and the position of the target object in the NMR image;

[0037] The feedback strength is determined according to the distance, wherein the distance is negatively correlated with the feedback strength;

[0038] The feedback intensity is provided to the user operating the terminal.

[0039] Optionally, before performing target identification on the MRI image, the method further includes:

[0040] Determine the three-dimensional model of the working end;

[0041] Based on the determined position of the working end in the NMR image, a three-dimensional model of the working end is generated.

[0042] This manual provides an obstacle avoidance warning device, including:

[0043] The first determining module is used to determine the actual position of the working end of the surgical device and the actual position of the human optical beacon based on the pose of the human optical beacon and the pose of the surgical device optical beacon collected by the tracker.

[0044] The acquisition module is used to acquire CT images collected during the operation, including the working end and the human optical beacon, as well as MRI images collected before the operation.

[0045] The second determining module is used to identify the working end and the human optical beacon in the CT image, and determine the first transpose matrix of the CT image and the tracker based on the actual position of the working end and the actual position of the human optical beacon;

[0046] The third determining module is used to determine the second transpose matrix of the CT image and the MRI image, and to determine the position of the working end in the MRI image based on the first transpose matrix and the second transpose matrix;

[0047] The identification module is used to identify targets in the NMR image and determine the location of the targets in the NMR image;

[0048] The display module is used to determine and display obstacle avoidance prompts based on the position of the working end and the position of the target object.

[0049] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described obstacle avoidance prompting method.

[0050] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement an obstacle avoidance prompting method.

[0051] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0052] The obstacle avoidance prompting method provided in this manual first determines the actual position of the surgical device's working end and the actual position of the human optical beacon based on the poses of the human optical beacon and the surgical device's optical beacon acquired by the tracker. It then acquires intraoperative CT images containing the working end and the human optical beacon, as well as preoperative MRI images. The working end and the human optical beacon are identified in the CT images, and a first transpose matrix between the CT image and the tracker is determined based on their actual positions. A second transpose matrix between the CT image and the MRI image is then determined, and the position of the working end in the MRI image is determined based on both the first and second transpose matrices. Target object identification is performed on the MRI image to determine the position of the target object. Finally, obstacle avoidance prompts are determined and displayed based on the positions of the working end and the target objects.

[0053] By registering CT images with data acquired by a tracking device, and then registering CT images with MRI images, the working end of the surgical equipment is mapped onto the MRI image, allowing the surgeon to observe the position of the working end in real time during the operation. The MRI image is then used to determine the location of the target object in the surgical area. Displaying the location of the target object along with the position of the working end helps the surgeon better plan the movement path of the working end, reducing the surgeon's workload and achieving a higher surgical completion rate. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0055] Figure 1 A flowchart illustrating an obstacle avoidance warning method provided in an embodiment of this specification;

[0056] Figure 2 A schematic diagram of the coordinate mapping of the working end provided in this manual;

[0057] Figure 3 A schematic diagram of the coordinate mapping of the working end provided in this manual;

[0058] Figure 4 A schematic diagram of the spine provided for this description;

[0059] Figure 5 This is a schematic diagram of an obstacle avoidance warning device provided in this specification;

[0060] Figure 6 This specification provides a corresponding Figure 1 A schematic diagram of the structure of an electronic device. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0062] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0063] Figure 1 A flowchart illustrating an obstacle avoidance warning method provided in this embodiment of the specification includes the following steps:

[0064] S100: Based on the pose of the human optical beacon and the pose of the surgical equipment optical beacon collected by the tracker, determine the actual position of the working end of the surgical equipment and the actual position of the human optical beacon.

[0065] The obstacle avoidance prompting process described in this specification typically involves data processing. In the embodiments described herein, the obstacle avoidance prompting process can be executed by a server. Of course, this specification does not limit the type of device or platform from which the obstacle avoidance prompting process can be implemented; for example, personal computers, intelligent surgical robots, and mobile terminals can also be used. For ease of description, the following description uses a server as the executing entity.

[0066] In this specification, the human optical beacon is fixed to the surgical area of ​​the patient's body, and the surgical equipment optical beacon is mounted on the surgical equipment. This facilitates the optical tracker in acquiring the poses of both the human optical beacon and the surgical equipment optical beacon.

[0067] In one or more embodiments of this specification, the server determines the actual position of the working end of the surgical device and the actual position of the human optical beacon based on the pose of the human optical beacon acquired by the optical tracker and the pose of the surgical device optical beacon. The surgical device can be a surgical tool such as the robotic arm of a surgical robot, and the surgical device optical beacon is located at the tail end of the surgical tool, not outside the surgical area of ​​the human body. The working end is the head of the surgical tool, such as the blade of a scalpel or the drill bit of a burr.

[0068] S102: Acquire intraoperative CT images containing the working end and the human optical beacon, and acquire preoperative MRI images.

[0069] In one or more embodiments of this specification, the server acquires intraoperative CT images including the working end and human optical beacons, as well as preoperative MRI images. Specifically, the server acquires three-dimensional CT images and three-dimensional MRI images. Of course, if the acquired CT images are two-dimensional, they can be converted into three-dimensional CT images.

[0070] S104: Identify the working end and the human optical beacon in the CT image, and determine the first transpose matrix of the CT image and the tracker based on the actual position of the working end and the actual position of the human optical beacon.

[0071] In one or more embodiments of this specification, the server identifies the working end and the human optical beacon in the CT image, and determines the first transpose matrix of the CT image and the tracker based on the actual position of the working end and the actual position of the human optical beacon.

[0072] Specifically, the server can use image recognition methods to determine the working end and human optical beacons in CT images. It can also determine the images of the working end and human optical beacons. Based on the determined image of the working end, the working end in the CT image is identified; based on the determined image of the human optical beacon, the human optical beacon in the CT image is identified.

[0073] Then, based on the positions of the working end and the human optical beacon in the CT image, determine the first coordinates of the working end and the human optical beacon in the coordinate system of the CT image. Based on the actual positions of the working end and the human optical beacon acquired by the optical tracker, the optical tracker displays the acquired actual positions in its three-dimensional coordinate system. The actual position of the working end is then the second coordinate of the working end in the three-dimensional coordinate system of the optical tracker, and the actual position of the human optical beacon is the second coordinate of the human optical beacon in the three-dimensional coordinate system of the optical tracker.

[0074] The second coordinates of the working end and the second coordinates of the human optical beacon are matched with the first coordinates of the working end and the first coordinates of the human optical beacon in the coordinate system of the 3D CT image. This allows for the determination of the first transpose matrix required for rotation, translation, and scaling when mapping the coordinates of the working end in the 3D coordinate system of the optical tracker to the coordinate system of the 3D CT image.

[0075] Figure 2 This is a schematic diagram of the coordinate mapping of the working end provided in this manual. Figure 2 The left side shows the three-dimensional coordinate system of the optical tracker, displayed in three-dimensional space, where the black dots represent the positions of the working end. Figure 2The right side of the image shows the three-dimensional coordinate system of the CT image, displayed in three-dimensional space, where the black dots represent the positions of the working end. The working end of the optical tracker in the three-dimensional coordinate system is mapped to the three-dimensional coordinate system of the CT image using the first transpose matrix.

[0076] S106: Determine the second transpose matrix of the CT image and the MRI image, and determine the position of the working end in the MRI image based on the first transpose matrix and the second transpose matrix.

[0077] In one or more embodiments of this specification, the server determines a second transpose matrix for registering CT images with MRI images, and determines the position of the working end in the MRI image based on the first transpose matrix and the second transpose matrix.

[0078] Specifically, the server identifies the surgical region in both the CT and MRI images. The surgical regions in both images are then segmented to obtain a first segmented image of the surgical region in the CT image and a second segmented image in the MRI image. For example, if the surgical region is a location on the spine, the first segmented image could be two vertebrae surrounding that location, and the second segmented image could also be two vertebrae surrounding that location. Image segmentation can be performed using neural network-based image segmentation models, clustering algorithms, watershed algorithms, region growing algorithms, etc.

[0079] The first and second segmented images of the surgical region are then registered using methods such as control point registration or supervised learning to determine the second transpose matrix of the CT and MRI images. The first and second segmented images represent the same area within the surgical region.

[0080] After determining the first transpose matrix and the second transpose matrix, the working end, which is determined based on the actual position collected by the optical tracker, can be multiplied by the first transpose matrix and the second transpose matrix to map the coordinates of the working end to the three-dimensional coordinate system where the NMR image is located, thereby determining the position of the working end in the NMR image.

[0081] Figure 3 This is a schematic diagram of the coordinate mapping of the working end provided in this manual. Figure 3 The left side shows the three-dimensional coordinate system of the optical tracker, displayed in three-dimensional space, where the black dots represent the positions of the working end. Figure 2 The center shows the three-dimensional coordinate system of the CT image, displayed in three-dimensional space, where the black dots represent the position of the working end. Figure 2The right side shows the three-dimensional coordinate system of the MRI image, displayed in three-dimensional space, where the black dots represent the position of the working end. The first transpose matrix maps the working end from the optical tracker's three-dimensional coordinate system to the CT image's three-dimensional coordinate system, and the second transpose matrix maps the working end from the CT image's three-dimensional coordinate system back to the MRI image.

[0082] S108: Perform target object identification on the NMR image to determine the location of the target object in the NMR image.

[0083] In one or more embodiments of this specification, the server performs target object identification on the NMR image to determine the location of the target object in the NMR image.

[0084] S110: Determine and display obstacle avoidance prompts based on the position of the working end and the position of the target object.

[0085] In one or more embodiments of this specification, the server determines and displays obstacle avoidance prompts based on the position of the working end and the position of the target object in the MRI image. These obstacle avoidance prompts are used to alert the doctor or surgical robot to avoid the target object.

[0086] based on Figure 1 This paper describes an obstacle avoidance prompting method. First, based on the poses of the human optical beacon and the surgical equipment optical beacon acquired by the tracker, the actual positions of the surgical equipment's working end and the human optical beacon are determined. Then, CT images containing the working end and the human optical beacon, acquired during surgery, and MRI images acquired preoperatively are acquired. The working end and the human optical beacon in the CT images are identified, and a first transpose matrix of the CT image and the tracker is determined based on their actual positions. A second transpose matrix of the CT image and the MRI image is determined, and the position of the working end in the MRI image is determined based on the first and second transpose matrices. Target object identification is performed on the MRI image to determine the position of the target object. Finally, obstacle avoidance prompts are determined and displayed based on the positions of the working end and the target objects.

[0087] By registering CT images with data acquired by a tracking device, and then registering CT images with MRI images, the working end of the surgical equipment is mapped onto the MRI images, allowing the surgeon to observe its position in real time during surgery. The MRI images are then used to determine the location of target objects in the surgical area. Displaying the target object's position along with the working end's position helps the surgeon better plan the equipment's path, reducing their workload and achieving a higher surgical completion rate. Furthermore, in subsequent surgeries, it is unnecessary to take more CT images for registration; the data acquired by the tracking device is sufficient to locate the working end of the surgical equipment and display it on the MRI images.

[0088] Furthermore, in one or more embodiments of this specification, the target object may be a nerve root. The server can perform nerve root identification on MRI images to determine the location of nerve roots in the MRI images.

[0089] Specifically, the MRI images are segmented to determine the location of nerve roots within the images. Image segmentation methods can include neural network-based image segmentation models, clustering algorithms, watershed algorithms, and region growing algorithms. Of course, the target object can also be cartilage tissue, vertebrae, etc.; this specification does not impose limitations, and the target object can be determined based on the actual situation.

[0090] In one or more embodiments of this specification, the server can determine the distance between the location of the working end and the location of the target object in the NMR image. When the distance is less than a preset threshold, obstacle avoidance prompts are determined and displayed therein, which are used to prompt avoidance of the target object.

[0091] For example, obstacle avoidance prompts can be indicated by highlighting the obstacle in red on the edge of the display screen of the control panel at the operating end.

[0092] In one or more embodiments of this specification, the server can determine the distance between the location of the working end and the location of the target object in the NMR image. Based on the distance, a feedback strength is determined, wherein distance and feedback strength are negatively correlated, i.e., the closer the distance, the greater the feedback strength. The feedback strength is then provided to the user operating the working end.

[0093] In one or more embodiments of this specification, the server can determine the distance between the position of the working end and the position of the target object in the NMR image, as the relative distance d. Then, a force feedback expression is determined. This force feedback expression has an initial feedback distance d1, used to begin generating resistance to the working end entering the region containing the target object; that is, when the relative distance d is less than the initial feedback distance d1, feedback force is generated. The force feedback expression also has a maximum feedback distance d2, used to generate maximum resistance when the working end is very close to the target object, alerting the user operating the working end to a higher level of danger. The force feedback expression is as follows:

[0094] F = G(max(d - d1, d2 - d1)) + F init

[0095] Among them, F init This can be either a preset constant force feedback or actual force feedback from the working end. G(x) is a function curve representing the change in force feedback within the interval d1 to d2. It is not limited to linear, exponential, or logarithmic curves that allow the force to vary with distance, and the force applied by this curve is only generated when moving towards the shortest distance from the target object. Of course, the target object can be a nerve root.

[0096] It's worth noting that in this manual, the server can be a computer or a monitoring device with a display screen in the operating room, used by the doctor to monitor the surgical process. It can also be a control console for the surgical robot. Of course, if the doctor needs to provide feedback on the force applied during manual surgery, the surgical tools used by the doctor can be modified so that the server can manipulate and provide feedback on the force applied.

[0097] In one or more embodiments of this specification, the server can determine a three-dimensional model of the surgical end. Based on the determined position of the surgical end in the MRI image, a three-dimensional model of the surgical end is generated in the MRI image. This allows users to more conveniently perform surgery using the surgical end through MRI images.

[0098] In one or more embodiments of this specification, the target object is a nerve root. The server can determine the surgical region in the MRI image. The surgical region is segmented to determine the location of the nerve root within the surgical region of the MRI image.

[0099] In one or more embodiments of this specification, the target object is a nerve root. The server can determine the surgical region in the MRI image. The surgical region is segmented to determine the location of the nerve root within the surgical region of the MRI image.

[0100] Next, determine the nerve roots within the preset range of the working end. Then, determine the distance between the position of the working end and the position of the nerve roots within the preset range. This is to facilitate subsequent steps; the specific details are described above and will not be repeated here.

[0101] In one or more embodiments of this specification, the target object is a nerve root. The server can perform image segmentation on the MRI image to determine the location of the nerve root in the MRI image.

[0102] Next, identify the nerve roots within a preset range on the working end. Then, determine the distance between the position of the working end and the positions of the nerve roots within the preset range. This is to facilitate subsequent steps; the specific details are described above and will not be repeated here. Alternatively, the nerve roots within the preset range on the working end can be determined without specifying them, and the distance between all identified nerve roots and the working end can be directly determined.

[0103] In one or more embodiments of this specification, the surgical area includes the spine, and the first segmented image and the second segmented image are two vertebrae at the same location of the spine.

[0104] The server determines the first coordinates of the first and second vertebrae in the first segmented image based on their positions. It then determines the third coordinates of the first and second vertebrae in the second segmented image based on their positions. Finally, based on the first and third coordinates of the first and second vertebrae, the server determines the second transpose matrix between the coordinate systems of the CT image and the MRI image.

[0105] Figure 4 This is a schematic diagram of the spine provided for this instruction. Figure 4 The solid rectangle represents the spine, the dashed rectangle represents the surgical area, and the two shaded areas represent the first and second vertebral bodies near the surgical area, respectively.

[0106] The optical tracker used in this specification may be an optical tracker from Northern Digital Inc. (NDI), or other infrared-based optical trackers. It may also be an RGB-D camera based on RealSense technology.

[0107] The above describes one or more embodiments of an obstacle avoidance warning method provided in this specification. Based on the same idea, this specification also provides a corresponding obstacle avoidance warning device, such as... Figure 5 As shown.

[0108] Figure 5 This is a schematic diagram of an obstacle avoidance warning device provided in this specification, specifically including:

[0109] The first determining module 500 is used to determine the actual position of the working end of the surgical device and the actual position of the human optical beacon based on the pose of the human optical beacon and the pose of the surgical device optical beacon collected by the tracker.

[0110] The acquisition module 502 is used to acquire CT images collected during the operation, including the working end and the human optical beacon, and to acquire MRI images collected before the operation.

[0111] The second determining module 504 is used to identify the working end and the human optical beacon in the CT image, and determine the first transpose matrix of the CT image and the tracker based on the actual position of the working end and the actual position of the human optical beacon.

[0112] The third determining module 506 is used to determine the second transpose matrix of the CT image and the MRI image, and to determine the position of the working end in the MRI image based on the first transpose matrix and the second transpose matrix;

[0113] The identification module 508 is used to identify targets in the NMR image and determine the location of the targets in the NMR image;

[0114] The display module 510 is used to determine and display obstacle avoidance prompts based on the position of the working end and the position of the target object.

[0115] Optionally, the second determining module 504 is specifically used to determine the working end and the human optical beacon in the CT image through an image recognition method; determine the first coordinates of the working end and the human optical beacon in the coordinate system of the CT image based on the positions of the working end and the human optical beacon in the CT image; determine the second coordinates of the working end and the human optical beacon in the coordinate system of the tracker based on the actual positions of the working end and the human optical beacon acquired by the tracker; and determine the first transpose matrix of the coordinate system of the CT image and the coordinate system of the tracker based on the first and second coordinates of the working end and the first and second coordinates of the human optical beacon.

[0116] Optionally, the third determining module 506 is specifically used to determine the surgical region in the CT image and the surgical region in the MRI image, perform image segmentation on the surgical region in the CT image and the surgical region in the MRI image to obtain a first segmented image of the surgical region in the CT image and a second segmented image of the surgical region in the MRI image, and determine a second transpose matrix of the CT image and the MRI image based on the first segmented image and the second segmented image, wherein the first segmented image and the second segmented image are segmented images of the same part of the surgical region.

[0117] Optionally, the surgical area includes the spine, and the first segmented image and the second segmented image are two vertebrae at the same location of the spine;

[0118] The third determining module 506 is further configured to determine the first coordinates of the first vertebra and the second vertebra in the first segmented image based on their positions in the first segmented image, determine the third coordinates of the first vertebra and the second vertebra in the second segmented image based on their positions in the second segmented image, and determine the second transpose matrix of the coordinate system of the CT image and the coordinate system of the MRI image based on the first and third coordinates of the first vertebra and the first and third coordinates of the second vertebra.

[0119] Optionally, the target object is a nerve root;

[0120] The identification module 508 is specifically used to determine the surgical region in the MRI image, segment the surgical region, and determine the location of the nerve root in the surgical region.

[0121] Optionally, the display module 510 is specifically used to determine the distance between the position of the working end and the position of the target object in the NMR image. When the distance is less than a preset threshold, obstacle avoidance prompt information is determined and displayed. The obstacle avoidance prompt information is used to prompt avoidance of the target object.

[0122] Optionally, the obstacle avoidance warning device further includes a feedback module 512;

[0123] The feedback module 512 is used to determine the distance between the position of the working end and the position of the target object in the NMR image, and to determine the feedback intensity according to the magnitude of the distance, wherein the distance is negatively correlated with the feedback intensity, and to provide the feedback intensity to the user operating the working end.

[0124] Optionally, the recognition module 508 is further configured to determine the three-dimensional model of the working end, and generate the three-dimensional model of the working end based on the determined position of the working end in the NMR image.

[0125] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This provides an obstacle avoidance prompt method.

[0126] This instruction manual also provides Figure 6 The diagram shows a schematic structural representation of the electronic device. Figure 6 As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 1 The obstacle avoidance prompting method described above.

[0127] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This provides an obstacle avoidance prompt method.

[0128] This instruction manual also provides Figure 6 The diagram shows a schematic structural representation of the electronic device. Figure 6As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 1 The obstacle avoidance prompting method described above.

[0129] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0130] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0131] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0132] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0133] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0134] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0140] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0145] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. An obstacle avoidance warning device, characterized in that, include: The first determining module is used to determine the actual position of the working end of the surgical device and the actual position of the human optical beacon based on the pose of the human optical beacon and the pose of the surgical device optical beacon collected by the tracker. The acquisition module is used to acquire CT images collected during the operation, including the working end and the human optical beacon, as well as MRI images collected before the operation. The second determining module is used to identify the working end and the human optical beacon in the CT image, and determine the first transpose matrix of the CT image and the tracker based on the actual position of the working end and the actual position of the human optical beacon; The third determining module is used to determine the second transpose matrix of the CT image and the MRI image, and to determine the position of the working end in the MRI image based on the first transpose matrix and the second transpose matrix; The identification module is used to identify targets in the NMR image and determine the location of the targets in the NMR image; The display module is used to determine and display obstacle avoidance prompts based on the position of the working end and the position of the target object.

2. The apparatus as claimed in claim 1, characterized in that, The second determining module is specifically used to determine the working end and the human optical beacon in the CT image using an image recognition method; determine the first coordinates of the working end and the human optical beacon in the coordinate system of the CT image based on their positions in the CT image; determine the second coordinates of the working end and the human optical beacon in the coordinate system of the tracker based on their actual positions acquired by the tracker; and determine the first transpose matrix of the coordinate system of the CT image and the coordinate system of the tracker based on the first and second coordinates of the working end and the first and second coordinates of the human optical beacon.

3. The apparatus as described in claim 1, characterized in that, The third determining module is specifically used to determine the surgical region in the CT image and the surgical region in the MRI image, perform image segmentation on the surgical region in the CT image and the surgical region in the MRI image to obtain a first segmented image of the surgical region in the CT image and a second segmented image of the surgical region in the MRI image, and determine a second transpose matrix of the CT image and the MRI image based on the first segmented image and the second segmented image, wherein the first segmented image and the second segmented image are segmented images of the same part of the surgical region.

4. The apparatus as described in claim 3, characterized in that, The surgical area includes the spine, and the first segmented image and the second segmented image are two vertebrae at the same location of the spine; The third determining module is further configured to determine the first coordinates of the first vertebra and the second vertebra in the first segmented image based on their positions in the first segmented image, determine the third coordinates of the first vertebra and the second vertebra in the second segmented image based on their positions in the second segmented image, and determine the second transpose matrix of the coordinate system of the CT image and the coordinate system of the MRI image based on the first and third coordinates of the first vertebra and the first and third coordinates of the second vertebra.

5. The apparatus as claimed in claim 1, characterized in that, The target object is a nerve root; The identification module is specifically used to determine the surgical region in the MRI image, segment the surgical region, and determine the location of the nerve root in the surgical region.

6. The apparatus as claimed in claim 1, characterized in that, The display module is specifically used to determine the distance between the position of the working end and the position of the target object in the NMR image. When the distance is less than a preset threshold, obstacle avoidance prompt information is determined and displayed. The obstacle avoidance prompt information is used to prompt avoidance of the target object.

7. The apparatus as claimed in claim 1, characterized in that, The obstacle avoidance warning device also includes a feedback module; The feedback module is used to determine the distance between the position of the working end and the position of the target object in the NMR image, and to determine the feedback intensity according to the magnitude of the distance, wherein the distance is negatively correlated with the feedback intensity, and to provide the feedback intensity to the user operating the working end.

8. The apparatus as claimed in claim 1, characterized in that, The identification module is further configured to determine the three-dimensional model of the working end, and generate the three-dimensional model of the working end based on the determined position of the working end in the NMR image.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the function of the device according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it performs the functions of the apparatus according to any one of claims 1 to 8.

Images