Method and apparatus for determining posture of surgical registration device

Abstract

A method, a surgical registration device (2), a sensor device (6), a system, a computer program, a computer-readable medium and a data carrier signal are disclosed. The surgical registration device (2) comprises a surface, an optical pattern (30) on the surface and an interface unit attached to a tracking system (4). The method comprises: obtaining (70) an image of the surgical registration device; detecting (72) at least one optical pattern in the image; obtaining (74) surface model data describing the surface; obtaining (76) a position of the optical pattern relative to the surface; determining (78) a second pose of the surface based on the detected optical pattern, the surface model data and the position of the optical pattern; obtaining (80) depth information describing the three-dimensional coordinates of the surface; and determining (82) a third pose of the surface of the surgical registration device based on the second pose, the surface model data and the depth information.

Description

Technical Field

[0001] The present disclosure generally relates to surgical navigation, that is, to a surgical registration device and a method for determining its posture. In particular, a method, a computer program, a computer readable medium, a data carrier signal, a processor, a sensor device, a surgical navigation system and a surgical registration device are disclosed. Background Art

[0002] Many surgical procedures benefit from determining the position and orientation of surgical objects, such as surgical instruments and patients, as a basis for providing navigation instructions to the surgeon. Surgical navigation typically requires initial registration of three-dimensional image data of the surgical object in a coordinate system of a surgical navigation system. During registration, the three-dimensional image data is arranged and oriented in the coordinate system relative to tracking markers associated with the surgical object (i.e., the patient or surgical instrument), which subsequently allows changes in the position and orientation of the three-dimensional image data in the coordinate system to be determined by tracking the movement of the tracking markers.

[0003] In an exemplary application of a surgical navigation system, a surgical instrument and a patient are each associated with a tracking marker, wherein three-dimensional image data previously acquired, for example, by a computed tomography (CT) scan, is registered with the patient tracking marker, and three-dimensional image data of the surgical instrument is registered with the instrument tracking marker. By tracking the tracking markers of the patient and the surgical instrument, the position and orientation of the two surgical objects (surgical instrument and patient), as well as the spatial relationship between the three-dimensional image data of the surgical instrument and the three-dimensional image data of the patient, can be determined. The determined spatial relationship can be displayed, for example, on a screen, thereby helping the surgeon to guide the surgical instrument relative to the patient.

[0004] During surgery, multiple tracking types (i.e., optical tracking, electromagnetic tracking, or ultrasonic tracking) can be used in parallel or sequentially. For example, an electromagnetic tracking system and an optical tracking system can be used in combination. In this case, the surgeon can select the tracking type that best suits the current task.

[0005] Sensor devices including image sensors and depth sensors have been described. For example, head-mounted augmented reality devices including cameras and time-of-flight cameras have been disclosed. Such sensor devices can be used as optical tracking systems in surgical navigation. Alternatively or additionally, such sensor devices can be equipped with a display to allow a magnified view of the surgical site superimposed with navigation instructions. In order for the magnified view to be accurate, the relative spatial relationship between the tracking system (e.g., an electromagnetic tracking system) of the surgical navigation system and the sensor device needs to be determined as accurately as possible.

[0006] A surgical registration device has been disclosed that is detectable by both an optical tracking system and an electromagnetic tracking system to allow the relative spatial relationship between the two tracking systems to be determined.

[0007] The aforementioned sensor devices are generally intended for consumer applications. Therefore, the accuracy of the position and orientation of the surgical registration device determined by such sensor devices is generally very low, which results in low tracking accuracy and low accuracy of the determined relative spatial relationship between the electromagnetic tracking system and the sensor device. Summary of the invention

[0008] What is needed is a technique for determining the pose of a surgical registration device that addresses one or more of the above-mentioned or other problems.

[0009] According to a first aspect, a computer-implemented method for determining a posture of a surgical registration device is provided, wherein the surgical registration device includes a surface and at least one optical pattern arranged on the surface. The method includes obtaining an image of at least a first portion of the surgical registration device in a first posture, and detecting at least one optical pattern in the image. The method also includes obtaining surface model data describing the surface of at least a second portion of the surgical registration device, and obtaining first position information indicating the position of at least one optical pattern relative to the surface of the surgical registration device. The method of the first aspect includes determining a second posture of the surface of the surgical registration device based on the detected at least one optical pattern, the surface model data, and the first position information. The method also includes obtaining depth information describing the three-dimensional coordinates of the surface of at least the second portion of the surgical registration device in the first posture, and determining a third posture of the surface of the surgical registration device based on the second posture, the surface model data, and the depth information.

[0010] The first portion of the surgical registration device may be equal to the second portion of the surgical registration device. In another variation, the first portion of the surgical registration device is different from the second portion of the surgical registration device. Alternatively, the first portion of the surgical registration device may at least partially overlap with the second portion of the surgical registration device, in other words, the first portion of the surgical registration device may include a portion that is also included in the second portion of the surgical registration device.

[0011] The first posture of the surgical registration device may be referred to as the actual posture of the surgical registration device. The first posture of the surgical registration device may be a first position and orientation of the surgical registration device in a given coordinate system. The second posture of the surgical registration device may be, for example, a second position and orientation of the surgical registration device in the given coordinate system. The third posture of the surgical registration device may be, for example, a third position and orientation of the surgical registration device in the given coordinate system. The given coordinate system may be an internal coordinate system of the sensor device, for example, a sensor device coordinate system.

[0012] The depth information is obtained, for example, by a sensor of a sensor device, such as a depth sensor including a time-of-flight camera or a stereo camera. The depth information may include depth pixels, or may be determined based on depth pixels obtained by a sensor of the sensor device. For example, the depth information describes a three-dimensional grid representing at least a portion of a surface of a surgical registration device (e.g., a surface of at least the second portion of the surgical registration device). The depth information may describe a point cloud, wherein each point defines a three-dimensional coordinate representing a point on the surface of the surgical registration device (e.g., a point on the surface of at least the second portion of the surgical registration device). In one example, the depth information describes a depth image including a plurality of depth image pixels. Each of the depth image pixels may include a depth value that describes the distance of an object depicted by each depth image pixel to a sensor of the sensor device.

[0013] The optical pattern is, for example, a two-dimensional pattern, for example, an arrangement of graphic symbols, barcodes, data matrix codes, Quick Response (QR) codes, PDF417 codes, MaxiCodes, AZTEC codes, etc. In one variant, the optical pattern is a pattern formed only in black and white, however, the optical pattern may include at least one element having a color.

[0014] Determining the third pose may include: determining, based on the second pose and based on the surface model data, a surface model describing a surface of at least a second portion of the surgical registration device in the second pose;

[0015] The surface model is matched to the depth information to determine the third pose.

[0016] For example, the surface model matches a portion of the depth information that satisfies at least one criterion selected from the following:

[0017] It does not represent predetermined surface properties;

[0018] does not represent objects other than surgical registration equipment; and

[0019] Portions of the surgical registration device having a predetermined color are not representative.

[0020] The predetermined surface attribute may be an edge, a convex portion, a concave portion, a planar portion of the surface of the surgical registration device, the planar portion being smaller than a predetermined minimum size; a portion of the surface of the surgical registration device that is hidden beneath another portion of the surgical registration device, etc. At least one of the predetermined surface attribute and the predetermined color may be specified in the surface model data, or may be specified in another data type obtained before matching the surface model to a portion of the depth information.

[0021] Determining the second posture may include extracting information from the at least one detected optical pattern. For example, the information indicates at least one parameter selected from the following: a size of the at least one optical pattern, a shape of the at least one optical pattern, identification data (ID) of the at least one optical pattern, a type of the at least one optical pattern, a position of the at least one optical pattern relative to a surface of a surgical registration device, a type of the surgical registration device, a shape of a surface of the surgical registration device, a shape of a surface of the second portion of the surgical registration device, and a known relative spatial relationship between the surgical registration device and a tracking system of the surgical navigation system. In one example, the information indicates another parameter.

[0022] In one variation, the surgical registration device includes at least one colored region disposed on a surface of the surgical registration device, wherein the at least one colored region is distinct from the at least one optical pattern, and wherein determining the second posture includes: detecting the at least one colored region in the image; obtaining second position information indicating a position of the at least one colored region relative to the surface of the surgical registration device; and

[0023] The second gesture is also determined based on the detected at least one colored area and the second position information.

[0024] For example, an initial second posture is determined based on the detected at least one optical pattern, the surface model data, and the first position information, and the initial second posture is corrected based on the detected at least one colored area and the second position information to determine the second posture. In another example, the second posture is determined in a single step based on the detected at least one optical pattern, the surface model data, the first timing information, the detected at least one colored area, and the second position information.

[0025] In one example, the surgical registration device includes at least one colored region disposed on a surface of the surgical registration device, wherein determining the third pose includes: obtaining second position information indicating a position of the at least one colored region relative to the surface of the surgical registration device; determining an approximate surface model based on the second pose, the surface model data, and the second position information, the approximate surface model describing an estimate of a surface of at least a second portion of the surgical registration device in the second pose as described by the depth information; and matching the approximate surface model with the depth information to determine the third pose. For example, the approximate surface model is matched with depth information obtained by a depth sensor of the sensor device.

[0026] Obtaining depth information may include: obtaining a plurality of depth pixels; obtaining a plurality of image pixels from an image; registering the plurality of image pixels to a plurality of depth pixels; and determining correction values ​​of the depth pixels based on color values ​​of the registered image pixels and based on values ​​of the registered depth pixels to determine the depth information. The depth information may include both values ​​of the (i.e., registered) depth pixels and correction values ​​of the depth pixels.

[0027] The approximate surface model is matched, for example, with the value of the depth pixel. In other words, the approximate surface model can be matched with the depth information including the value of the depth pixel. That is, the approximate surface model is matched, for example, with the measured depth value obtained by the depth sensor as the depth information. On the other hand, the surface model can be matched with the corrected value of the depth pixel. In other words, the surface model can be matched with the depth information including the corrected value of the depth pixel. That is, the surface model is matched, for example, with the color-adjusted depth value obtained by correcting the measured depth value.

[0028] The second pose and the third pose may each describe a relative spatial relationship between the surgical registration device and a sensor device including at least one sensor configured to obtain at least one data selected from depth information and an image.

[0029] For example, the method further includes obtaining a known relative spatial relationship between the surgical registration device and a tracking system of the surgical navigation system, and determining a relative spatial relationship between the sensor device and the tracking system based on the third posture and the known relative spatial relationship.

[0030] The sensor device includes, for example, a camera configured to obtain an image and a depth sensor configured to obtain depth information. The sensor device may be an augmented reality device including a camera and a depth sensor. The tracking system of the surgical navigation system is, for example, an optical or electromagnetic tracking system configured to track at least one tracking marker attached to a patient or a surgical instrument.

[0031] According to a second aspect, a surgical registration device for surgical navigation is provided. The surgical registration device comprises a surface, wherein the surface of at least a second portion of the surgical registration device is configured to be described by depth information and described by surface model data. The surgical registration device further comprises: at least one optical pattern arranged at a predetermined position on the surface of the surgical registration device and configured to be depicted in an image of at least the first portion of the surgical registration device; and an interface unit configured to removably attach the surgical registration device to a tracking system of a surgical navigation system in a predetermined spatial relationship.

[0032] The surface is, for example, configured such that at least two portions of the surface of the surgical registration device can be seen in the viewing direction regardless of the orientation of the surgical registration device relative to the viewing direction. In one example, the at least two portions are located in different planes, and the different planes are, for example, not parallel to each other. Alternatively, the at least two portions of the surface can be located on different portions of a spherical portion of the surface.

[0033] At least one optical pattern may include a plurality of optical patterns. For example, each of the plurality of optical patterns is different. The plurality of optical patterns may include or consist of optical patterns that are distinguishable from each other. In this case, each of the plurality of optical patterns is, for example, arranged on a different planar portion of the surface. In one example, the different planar portions of the surface are not parallel to each other.

[0034] In one variation, the surgical registration device includes at least one colored area on the surface of the surgical registration device, wherein the at least one colored area is different from the at least one optical pattern. The colored area can be a geometric form colored in a single color, such as a circle, an oval, a triangle, a rectangle, a pentagon, a star, a company logo, a letter, etc.

[0035] In another variant, at least one colored region and at least one optical pattern overlap each other, for example, at least partially overlap. In this case, at least one colored region may be equal to at least one optical pattern.

[0036] In one example, at least one colored region includes a plurality of colored regions. For example, each of the plurality of colored regions is different. The plurality of colored regions may include or consist of colored regions that are distinguishable from each other. Each of the plurality of colored regions may be arranged on a different planar portion of the surface of the surgical registration device. In one example, the different planar portions of the surface are not parallel to each other.

[0037] The surgical registration device is designed, for example, as a re-sterilizable component.

[0038] According to a third aspect, a processor is provided. The processor is configured to execute the method described in the first aspect. Specifically, the processor is configured to: obtain an image of at least a first portion of a surgical registration device in a first posture; detect at least one optical pattern in the image; obtain surface model data describing a surface of at least a second portion of the surgical registration device; obtain first position information indicating a position of at least one optical pattern relative to the surface of the surgical registration device; and determine a second posture of the surface of the surgical registration device based on the detected at least one optical pattern, the surface model data, and the first position information.

[0039] The processor is further configured to obtain depth information describing three-dimensional coordinates of a surface of at least a second portion of the surgical registration device in the first pose and determine a third pose of the surface of the surgical registration device based on the second pose, the surface model data, and the depth information.

[0040] According to a fourth aspect, a sensor device is provided. The sensor device includes an image sensor configured to acquire an image and a sensor configured to acquire at least one of depth information and depth pixels. The sensor device also includes the processor described in the third aspect. For example, the sensor device includes a camera as an image sensor. As a sensor configured to acquire at least one of depth information and depth pixels, the sensor device may include a depth sensor. The depth sensor may include a stereo camera. In this case, one of the cameras of the stereo camera may be used as an image sensor. Alternatively, the sensor device may include different depth sensors, such as a light detection and ranging (LIDAR) sensor, a time of flight (TOF) sensor, and the like. The sensor device may also include a display configured to display navigation information based on at least a third posture. The sensor device may be configured to track at least one tracking marker, for example, using a stereo camera included in the sensor device. The sensor device may be an augmented reality device, for example, a head mounted display (HMD) such as HOLOLENS® or MAGIC LEAP ONE®.

[0041] According to a fifth aspect, a surgical navigation system is provided, comprising a surgical registration device according to the second aspect and a tracking system configured to track at least one tracking marker attached to a patient or a surgical instrument, wherein the surgical registration device is attached to the tracking system in a predetermined spatial relationship via an interface unit. The tracking system of the surgical navigation system may be an optical or electromagnetic tracking system configured to track at least one of the patient and the surgical instrument, for example by tracking at least one tracking marker attached to the patient or the surgical instrument.

[0042] According to a sixth aspect, a computer program is provided. The computer program comprises instructions, which, when executed by a processor, cause the processor to perform the method according to the first aspect. Alternatively or additionally, a computer program product may be provided, wherein the computer program product comprises instructions, which, when executed by a computer, cause the computer to perform the method steps described in the first aspect.

[0043] According to a seventh aspect, there is provided a computer readable storage medium having stored thereon a computer program or a computer program product according to the sixth aspect.

[0044] According to an eighth aspect, there is provided a data carrier signal carrying a computer program or a computer program product according to the sixth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other details, advantages and aspects of the present invention will become apparent from the following embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 An arrangement comprising a surgical registration device, a tracking system and a sensor device is shown;

[0047] Figure 2 shows a front view of a first example of a surgical registration apparatus;

[0048] Figure 3 shows a side view of a first example of a surgical registration device disposed on a tracking system;

[0049] Figure 4 showing a perspective view of a second example of a surgical registration apparatus;

[0050] Figure 5 shows a top view of a second example of a surgical registration apparatus;

[0051] Figure 6 shows a side view of a second example of a surgical registration device disposed on a tracking system;

[0052] Figure 7 A third example of a surgical registration device is shown;

[0053] Figure 8 A fourth example of a surgical registration apparatus is shown;

[0054] Fig. 9 A fifth example of a surgical registration apparatus is shown;

[0055] Fig.10 A flowchart showing the method described herein; and

[0056] Fig.11 Shows Fig.10 Another flow chart of the method. DETAILED DESCRIPTION

[0057] In the following description, exemplary embodiments of a surgical registration device, a processor, a computer program, a computer-readable storage medium, a data carrier signal, a sensor device, a surgical navigation system, and a method for determining a posture of a surgical registration device will be described with reference to the accompanying drawings. The same reference numerals will be used to represent the same or similar structural features or method steps.

[0058] Figure 1An arrangement 100, also referred to as system 100, is shown, which comprises a surgical registration device 2, a tracking system 4 and a sensor device 6. In the illustrated arrangement 100, the tracking system 4 is an electromagnetic tracking system. Of course, other types of tracking systems may be used as tracking system 4, for example an optical tracking system or an ultrasound tracking system.

[0059] The first tracking marker 8 is attached to the patient's body 10. The tracking system 4 comprises an electromagnetic field generator 12 and a positioning unit 14. The electromagnetic field generator 12 is configured to generate an electromagnetic field, which extends over at least a portion of the body 10. The first tracking marker 8 is an electromagnetic tracking marker. The first tracking marker 8 is configured to sense the generated electromagnetic field and provide measurement data to the positioning unit 14. Based on the measurement data of the first tracking marker 8, the positioning unit 14 determines the position and orientation of the first tracking marker 8 in the tracking coordinate system 16. This enables the tracking system 4 to track the body 10.

[0060] In the case where the tracking system 4 is an optical tracking system, the surgical registration device 2 is removably attached to an optical detection unit (not shown) of the tracking system 4 in a known spatial relationship. Such an optical detection unit is, for example, a stereo camera. In this case, the first tracking marker 8 is an optical tracking marker that can be tracked by the optical tracking system to determine the position and orientation of the first tracking marker 8 in the tracking coordinate system 16.

[0061] Of course, two tracking systems may be provided, for example, the optical tracking system and the electromagnetic tracking system described above. In this case, one or more tracking markers attached to the body 10 may be provided for the electromagnetic tracking system, and one or more tracking markers attached to the body 10 may be provided for the optical tracking system.

[0062] exist Figure 1 In the arrangement 100 shown, the surgical registration device 2 is removably attached to the electromagnetic field generator 12 in a known spatial relationship via an interface unit included in the surgical registration device 2 (not shown). If the tracking system 4 is an optical tracking system, the surgical registration device 2 can be removably attached to the optical detection unit of the optical tracking system in a known spatial relationship via the interface unit. This means that the pose (position and orientation) of the surgical registration device 2 in the tracking coordinate system 16 is known. In another variant, the surgical registration device 2 is arranged in a fixed spatial relationship relative to a third tracking marker (not shown) tracked by the tracking system 4. By determining the position and orientation of the third tracking marker in the tracking coordinate system 16, the pose (at least one of the position and orientation) of the surgical registration device 2 in the tracking coordinate system 16 can be determined.

[0063] The sensor device 6 includes a camera 18 as an image sensor and a depth sensor 20 as a sensor configured to acquire at least one of depth information and depth pixels. A processor 22 is connected to the sensor device 6. The processor 22 may be arranged in the sensor device 6.

[0064] Optionally, a second tracking marker 24 may be arranged on the subject 10. The second tracking marker 24 is an optical tracking marker. Based on the image obtained by the camera 18, or based on the stereoscopic image of a stereo camera (not shown) included in the sensor device 6, the posture of the second tracking marker 24 can be determined by the processor 22 in the sensor device coordinate system 26, so that the sensor device 6 can directly track the patient. However, the provision of the second tracking marker 24 is not necessary, because the posture of the surgical registration device 2 can be determined in the sensor device coordinate system 24, as will be described in detail below, wherein the posture of the surgical registration device 2 in the tracking coordinate system 16 is known. In other words, the registration between the sensor device coordinate system 24 and the tracking device coordinate system 16 can be derived, so that the tracking posture of the patient in the tracking coordinate system 16 can be converted into a posture in the sensor device coordinate system 24, so that the sensor device 6 can indirectly track the patient via the tracking system 4.

[0065] The surgical registration device 2 includes a body 28 having a surface and at least one optical pattern 30 arranged on the surface. The at least one optical pattern 30 is, for example, a two-dimensional black and white QR code. The at least one optical pattern 30 is arranged at a predetermined position on the surface. This means that the position and orientation of the at least one optical pattern 30 in the tracking coordinate system 16 are known. The characteristics of the at least one optical pattern 30 (e.g., geometry, pattern type, size, etc.) are also known. One or more of these characteristics may be specific to an optical pattern and therefore differ between different optical patterns. At least one optical pattern in an image may be detected based at least in part on one or more of these characteristics.

[0066] The surgical registration device 2 includes at least one colored area arranged on a surface (not shown). The colored area is, for example, a circle with a single color. The at least one colored area is arranged at a predetermined position on the surface. This means that the position and orientation of the at least one colored area in the tracking coordinate system 16 are known. The characteristics (e.g., geometry, color, size, etc.) of the at least one colored area are also known. One or more of these characteristics may be specific to a given colored area and therefore differ between different colored areas. At least one colored area in the image may be detected based at least in part on one or more of these characteristics.

[0067] The processor 22 is configured to determine a pose of the surgical registration device 2 in the sensor device coordinate system 26. Subsequently, a transformation between the sensor device coordinate system 26 and the tracking coordinate system 16 may be determined by the processor 22, since the pose of the surgical registration device 2 is known in both the tracking coordinate system 16 and the sensor device coordinate system 26. In the illustrated arrangement 100, the processor 22 is connected (i.e., via a wireless data connection) to the tracking system 4. The processor 22 may be configured to determine navigation information to be displayed to the surgeon based on data received from the sensor device 6 and the tracking system 4. The processor 22 may further control the tracking system 4, for example by controlling the generation of an electromagnetic field by the electromagnetic field generator 12.

[0068] Figure 2 Shows Figure 1 A front view of a first example of a surgical alignment device 2. It can be seen that the main body 28 is formed by a plurality of cubes of different sizes. Of course, the main body 28 can be formed as a solid so that the cubes do not need to be connected to each other. In other words, the reference to the cube may only refer to the form of the surface of the surgical alignment device 2. A plurality of small cubes 32 form a base plate 34, an intermediate plate 37 and a side plate 39. Two large cubes 36 are arranged on the base plate 34 adjacent to the side plate 39 and the intermediate plate 37. At least one optical pattern 30 is provided on each visible plane side of each large cube 36. Figure 2 In FIG. 2 , only two optical patterns 30 are depicted due to the selection of the front view of the surgical registration device 2. On the substrate 34, a convex spherical portion 40 is provided, which is in contact with a large cube 36, the substrate 34 and one of the intermediate plate 37 or the side plate 39, respectively. The surface of the surgical registration device 2 is mirror-symmetrical to the virtual plane 38, which divides the intermediate plate 37 into two similar parts. The optical patterns 30 are not mirror-symmetrical to each other. Preferably, the optical patterns 30 are different from each other. In this example, the overall form of the surface of the surgical registration device 2 can be roughly described as a cone cut in half.

[0069] The surface of the surgical registration device 2 is configured so that regardless of the orientation of the surgical registration device 2 relative to the viewing direction, at least two parts of the surface of the surgical registration device 2 can be seen in the viewing direction, wherein the at least two parts are located in different planes. For example, the surfaces of the small cubes 32 forming the substrate 34 are located in different planes. This results in different distances between the surface of the surgical registration device 2 and the depth sensor 20, which is independent of the relative posture of the surgical registration device 2 relative to the depth sensor 20. In other words, different depth pixel values ​​can be obtained using the depth sensor 20, which describes the distance of the surface of the surgical registration device 2 to the depth sensor 20. Advantageously, the depth sensor 20 can detect many contours of the surface, not just one plane.

[0070] Figure 3 Shown is the layout Figure 1 4. The surgical registration device 2 is removably attached to the electromagnetic field generator 12 of the tracking system 4 such that the base plate 34 is arranged on top of the electromagnetic field generator. The surgical registration device 2 is removably connected to the electromagnetic field generator 12 in a fixed spatial relationship via an interface unit (not shown). In this example, the size of the surgical registration device 2 is determined so that it does cover the entire width of the electromagnetic field generator 12. The side of the side plate 39 facing away from the large cube 36 is aligned with a side of the electromagnetic field generator 12. In this example, the electromagnetic field generator 12 with the attached surgical registration device 2 is arranged so that the camera 18 of the sensor device 6 can detect at least one optical pattern 30.

[0071] Figure 4 Shows Figure 1 A perspective view of a second example of a surgical registration device 2. The surgical registration device of this example also includes the surgical registration device already referred to Figure 2 and Figure 3 Several components described in the present invention are described, so the description of some of them is omitted here. Unlike the first example of the surgical registration device 2, in the second example, the overall form of the surface of the surgical registration device 2 can be roughly described as a cone. In the second example, the body 28 of the surgical registration device is formed by four large cubes 36 instead of two and a plurality of small cubes 32. The surface of the surgical registration device 2 is mirror-symmetrical with the virtual plane 38, which divides the middle plate 37 into two similar parts. The surface of the surgical registration device 2 is also mirror-symmetrical with the virtual plane 42, which divides the side plate 39 into two similar parts. It can be seen that at least one optical pattern 30 includes a plurality of (a total of twelve) optical patterns 30. Each of the plurality of optical patterns 30 is arranged on a different planar portion of the surface. The different planar portions of the surface are not all parallel to each other. The optical patterns 30 are not only arranged on different planar portions of the surface, but also on different sides of the large cube 36, which are not parallel to each other. Although the optical patterns 30 shown are the same, they can also be different from each other.

[0072] Figure 5 The above reference Figure 4 A top view of a second example of a surgical registration device 2 is depicted. The optical patterns 30 are not mirror-symmetrical to each other. Instead, the optical patterns 30 have the same orientation but different positions on the surface of the surgical registration device 2. Note that the optical patterns 30 are preferably different from each other, for example, representing different QR codes.

[0073] Figure 6 Shown is the layout Figure 1A side view of a second example of a surgical registration device 2 on an electromagnetic field generator 12 of a tracking system 4. In addition, in this case, the surgical registration device 2 can be arranged on the electromagnetic field generator 12 via an interface unit (not shown). In the example shown, the surgical registration device 2 is arranged on the front side of the electromagnetic field generator 12 instead of on the front side as shown in FIG. Figure 3 Of course, the surgical registration device 2 can be arranged on another surface of the electromagnetic field generator 12 as long as the relative position between the two devices is known. Preferably, the surgical registration device 2 is arranged on the electromagnetic field generator 12 so that the camera 18 and the depth sensor 20 of the sensor device 6 can detect most of the surface of the surgical registration device 2.

[0074] Figure 7 Shows Figure 1 The third example of the surgical registration device 2. The surgical registration device 2 of this example also includes the Figures 2 to 6 The present invention relates to a surgical registration device 2. ...

[0075] Figure 8 Shows Figure 1 The fourth example of the surgical registration device 2. The surgical registration device of this example also includes the Figures 2 to 7 The surface of the surgical registration device 2 of the fourth example is Figure 7 The surfaces of the surgical registration device 2 of the third example shown are identical. Each of the three visible planar sides of the large cube 36 is covered with an optical pattern 30. On the surface of the surgical registration device 2 of the fourth example, at least one colored area is arranged. Figure 8 As can be seen in , at least one colored area includes a plurality of colored areas arranged on the surface of the surgical registration device 2. At least one colored area is different from at least one optical pattern 30. A first colored area 43 having a first color is arranged on the top side of the small cube 32. A second colored area 44 having a second color is arranged on the right side of the small cube 32. A third colored area 46 having a third color is arranged on the left side of the small cube 32. Therefore, the first colored area 43, the second colored area 44, and the third colored area 46 are arranged on different planar portions of the surface of the surgical registration device 2, wherein the different planar portions of the surface are not all parallel to each other. The first color, the second color, and the third color may be different from each other. The surgical registration device 2 of the fourth example may be arranged on Figure 1 The electromagnetic field generator 12, as shown above Figure 3 and Figure 6 described.

[0076] Fig. 9 Shows Figure 1 The fifth example of the surgical registration device 2. The surgical registration device of this example also includes the Figures 2 to 8 Several components are described, and descriptions of some of them are omitted here. In the fifth example of the surgical registration device 2, the main body 28 of the surgical registration device 2 is formed by a large cube 36 and a small cube 32. The optical pattern 30 is arranged on the left side of the small cube 32. The second colored area 44 having the second color is arranged on the right side of the small cube 32. The third colored area 46 having the third color is arranged on the left side of the large cube 36. In addition, the surgical registration device 2 of the fifth example can be arranged on Figure 1 The electromagnetic field generator 12, as shown above Figure 3 and Figure 6 described.

[0077] Figures 1 to 8 The surgical registration device 2 is constructed as a component that can be repeatedly sterilized. For example, the surgical registration device 2 is formed as a solid component. The body 28 of the surgical registration device 2 can be formed as a rigid body, for example, a molded component. The surgical registration device may not include movable parts. The edges of the surface of the surgical registration device 2 are, for example, rounded or blunt. The surgical registration device is made of a material that is resistant to repeated sterilization. At least one optical pattern 30 and at least one colored area are permanently formed on the surface of the surgical registration device so that the at least one optical pattern 30 and the at least one colored area can still be detected after the sterilization of the surgical registration device 2. For example, the at least one optical pattern 30 and the at least one colored area are covered by a protective transparent component of the surgical registration device 2, which is rigidly connected to the body 28 and protects the at least one optical pattern 30 and the at least one colored area from being removed. Alternatively or additionally, at least one optical pattern 30 and at least one colored area are permanently printed on the surface of the surgical registration device 2.

[0078] Of course, the above reference Figures 2 to 9 The features of one of the examples of the described surgical registration device 2 can be combined with other examples described with reference to these figures. For example, the surgical registration device 2 of the first example can include the second colored area 44 of the fourth example, the surgical registration device of the third example can be formed into a cone shape with double mirror symmetry, as described with reference to the second example, and so on.

[0079] Next, we will refer to Fig.10 The method according to the present disclosure is described. The method can be Figure 1The method may be performed by the processor 22 included in the sensor device 6, the sensor device 6 also including the camera 18 and the depth sensor 20. The method may be performed by a system including the processor 22 and the surgical registration device 2. The method may be performed by a system including the sensor device 6, the sensor device 6 including the processor 22, wherein the system also includes at least the surgical registration device 2.

[0080] In step 70, an image of at least a first portion of the surgical registration device 2 in a first pose is obtained. At least the first portion includes at least one optical pattern 30 arranged on a surface of the surgical registration device 2. The first pose is a first position and orientation of the surgical registration device in the sensor device coordinate system 26. The first pose may also be referred to as a true, precise, or correct pose of the surgical registration device 2 in the sensor device coordinate system 26.

[0081] In step 72, at least one optical pattern 30 is detected in the image. A pattern detection algorithm may be used to detect the optical pattern 30 in the image. For example, it is known in advance that the type of the optical pattern 30 is a black and white QR code, and the pattern detection algorithm searches for any black and white QR code in the acquired image. In the case where the type of the optical pattern 30 is not known in advance, the pattern detection algorithm may search for different types of patterns such as barcodes, colored QR codes, black and white QR codes, etc.

[0082] The processor 22 may extract information from the detected optical pattern 30. The information indicates, for example, at least one parameter selected from the following: the size of at least one optical pattern 30, the shape of at least one optical pattern 30, the ID of at least one optical pattern 30, the type of at least one optical pattern 30, the position of at least one optical pattern 30 relative to the surface of the surgical registration device 2 (e.g., in the model coordinate system), the type of the surgical registration device 2, the shape of the surface of the surgical registration device 2, the shape of the surface of the second portion of the surgical registration device 2, and a known relative spatial relationship between the surgical registration device 2 and the tracking system 4 (e.g., between the surgical registration device 2 and the electromagnetic field generator 12).

[0083] Based on the detection of at least one optical pattern 30 in the image, the processor 22 determines, for example, using a homography algorithm, the position and orientation of the detected optical pattern 30 in the sensor device coordinate system 26. For example, it is known that the actual size and shape of the detected optical pattern 30 on the surface of the surgical registration device 2 is known as information from the detected optical pattern 30, or is extracted as information from the detected optical pattern 30. Based on the size and shape of the optical pattern 30 detected in the image and the actual size and shape of the optical pattern 30 on the surface of the surgical registration device 2, the processor 22 can determine the position and orientation of the detected optical pattern 30 in the sensor device coordinate system 26.

[0084] In step 74, surface model data is obtained. The surface model data describes the surface of at least the second part of the surgical registration device 2. The surface of the second part of the surgical registration device 2 is a portion of the surface of the surgical registration device 2. The second part may be different from the first part, may be partially identical to the first part, or may be equal to the first part. For example, the surface model is an arithmetic definition of the surface of at least the second part, a three-dimensional mesh representing the surface of at least the second part, a point cloud representing the surface of at least the second part, or three-dimensional image data of at least the second part. The surface model data describes the surface of at least the second part of the surgical registration device 2 in a model coordinate system.

[0085] In step 76, first position information indicating the position of the at least one optical pattern 30 relative to the surface of the surgical registration device 2 is obtained. The first position information indicates the position of the at least one optical pattern 30 as a position and orientation in the model coordinate system. Alternatively, the first position information may indicate the position of the at least one optical pattern 30 by defining a characteristic of the surface at the position of the at least one optical pattern 30. In this case, the first position information may define that the at least one optical pattern 30 is arranged on a planar portion of the surface, on a rectangular plane of the surface, on a spherical portion of the surface, on a plane adjacent to a large cube 36 forming the surface, etc.

[0086] In step 78, a second pose of the surface of the surgical registration device 2 is determined based on the detected at least one optical pattern 30, the surface model data, and the first position information. Specifically, as described above, the position and orientation of the detected at least one optical pattern 30 are determined in the sensor device coordinate system 26. Since the position of the detected at least one optical pattern 30 on the surface of the surgical registration device 2 is known according to the first position information, and since the surface of the surgical registration device 2 is known according to the surface model data, the processor 22 can determine the second pose of the surface of the surgical registration device 2. The second pose is a second position and orientation of the surface of the surgical registration device 2 in the sensor device coordinate system 26. Therefore, the second pose describes the relative spatial relationship between the surgical registration device 2 and the sensor device 6. The second pose can be referred to as an optically detected pose, a pose obtained by analyzing an image, an image-based pose, etc.

[0087] As described above, at least one of the colored regions may be different from at least one of the optical patterns 30. In this case, different positions and orientations of the colored regions and the optical patterns 30 may be used to improve the accuracy of the second gesture.

[0088] Specifically, determining the second posture may include detecting at least one colored region in the image. The at least one colored region may be detected in the image based on a pattern detection algorithm, a color filter applied to the image, searching the image for pixels having color values ​​within a predetermined range, etc. Based on known characteristics of the at least one colored region (e.g., the size and shape of the at least one colored region), the processor 22 may determine the position and orientation of the detected at least one colored region in the sensor device coordinate system 26, for example, using a homography algorithm.

[0089] Then, determining the second posture may also include obtaining second position information indicating the position of the at least one colored area relative to the surface of the surgical registration device 2. The second position information indicates the position of the at least one colored area as a position and orientation in the model coordinate system. Alternatively, the second position information may indicate the position of the at least one colored area by defining the characteristics of the surface at the position of the at least one colored area. In this case, the second position information may define that the at least one colored area is arranged on a planar portion of the surface, on a rectangular plane of the surface, on a spherical portion of the surface, on a plane adjacent to a large cube 36 forming the surface, etc.

[0090] A second gesture may then also be determined based on the detected at least one colored area and the second position information.

[0091] In one variation, an initial second pose is determined based on the detected at least one colored region, the second position information, and the surface model data. Specifically, as described above, the position and orientation of the detected at least one colored region are determined in the sensor device coordinate system 26. Since the position of the detected at least one colored region on the surface of the surgical registration device 2 is known according to the second position information, and since the surface of the surgical registration device 2 is known according to the surface model data, the processor 22 can determine the initial second pose of the surface of the surgical registration device 2. The initial second pose is the initial position and orientation of the surface of the surgical registration device 2 in the sensor device coordinate system 26. The initial second pose can then be corrected based on the detected at least one optical pattern 30 and the first position information to determine the second pose.

[0092] In another variation, an initial second posture is determined based on the detected at least one optical pattern 30, the first position information and the surface model data, and then the initial second posture is corrected based on the detected at least one colored area and the second position information to determine the second posture.

[0093] In another variation, the initial second posture is not determined. In this case, the second posture can be determined based on the detected at least one optical pattern 30, the first position information, the detected at least one colored area, the second position information and the surface model. For example, the posture determined by the detected at least one optical pattern 30 in the sensor device coordinate system 26 and the posture determined by the at least one colored area in the sensor device coordinate system 26 can be matched with the surface described by the surface model data using a point-to-plane matching algorithm or a face-to-point matching algorithm. As a result of the matching, the second posture of the surface of the surgical registration device 2 can be obtained.

[0094] In step 80, depth information describing the three-dimensional coordinates of the surface of at least the second part of the surgical registration device 2 in the first pose is obtained. The three-dimensional coordinates of the surface of at least the second part of the surgical registration device 2 are described in the sensor device coordinate system 26 by the depth information. The depth information can be a depth image of the at least the second part of the surgical registration device 2, a three-dimensional mesh representing the surface of at least the second part of the surgical registration device 2, a point cloud representing points located on the surface of at least the second part of the surgical registration device 2, etc. The first pose is a first position and orientation of the surgical registration device 2 in the sensor device coordinate system 26, as is the image obtained. For example, the depth information is obtained by the depth sensor 20, and the image is obtained by the camera 18 at the same time. Alternatively, the depth information and the image are obtained at different points in time when the surgical registration device 2 has a specific spatial relationship with the sensor device 6 (which relationship can be described by the first pose).

[0095] The measurements of a depth sensor are often affected by the color of the object being detected. Depending on the color of the object being detected, it may be determined to be closer or farther away from the depth sensor 20 than it actually is. In other words, because the depth sensor 20 is sensitive to color, the depth pixels may not describe the correct three-dimensional coordinates. Given this assertion, obtaining depth information may include several steps.

[0096] In one step, a plurality of depth pixels may be obtained. Each depth pixel describes the three-dimensional coordinates of a single point or a single area on the surface of at least the second portion of the surgical registration device 2. These three-dimensional coordinates are described in the sensor device coordinate system 26. The depth pixel may be obtained using the depth sensor 20 of the sensor device 6. For example, the depth sensor 20 is a TOF camera that obtains a depth image, wherein each pixel included in the depth image is a depth pixel. The depth sensor 20 may alternatively include a stereo camera that acquires two images. The depth sensor 20 or the processor 22 may then determine a depth image (also referred to as a parallax image) based on the image acquired by the stereo camera. In addition, in this case, each pixel included in the depth image is a depth pixel. Of course, the depth pixel may describe the three-dimensional coordinate in a depth sensor coordinate system different from the sensor device coordinate system 26. The coordinates may then be converted from the depth sensor coordinate system to the sensor device coordinate system 26 using a known conversion between the two coordinate systems.

[0097] In another step, a plurality of image pixels may be obtained from the image. In the case where the image is a color image, each of the image pixels has a color value. The image pixels may define the color value of the pixel in the sensor device coordinate system 26. Of course, the image pixels may describe the color value in a camera coordinate system that is different from the sensor device coordinate system 26. The coordinates may then be converted from the camera coordinate system to the sensor device coordinate system 26 using known transformations between the two coordinate systems.

[0098] In a subsequent step, the plurality of image pixels may be registered to the plurality of depth pixels. One or more known transformations between the camera coordinate system, the depth sensor coordinate system, and the sensor device coordinate system 26 may be used to transform the plurality of image pixels and the plurality of depth pixels into a common reference system (e.g., the sensor device coordinate system 26). The registration of the plurality of image pixels with the plurality of depth pixels may be referred to as inter-camera registration. As a result of the registration, a registered depth image may be obtained, wherein each depth pixel is registered with an image pixel having a specific color value (i.e., the associated image pixel).

[0099] In another step, the depth information can be determined by determining a correction value for the depth pixel based on the color value of the registered image pixel and based on the value of the registered depth pixel. Predetermined offsets associated with different color values ​​can be used for this purpose. These offsets can be obtained by the processor 22 as part of the depth information, the surface model data or as separate data.

[0100] For example, a first depth pixel has a value {x1, y1, z1} that describes the three-dimensional coordinates of a point located on the surface of the surgical registration device 2 in the sensor device coordinate system 26. The first depth pixel is then registered to a first image pixel having a color value {R1, Y1, B1} by the processor 22. The processor 22 may then select a predetermined offset {a1, b1, c1} associated with the color value {R1, Y1, B1} from the data obtained by the processor 22. The processor 22 may then apply the offset {a1, b1, c1} to the value {x1, y1, z1} of the first depth pixel to determine a corrected value {x1 + a1, y1 + b1, z1 + c1} of the first depth pixel.

[0101] In another example, the predetermined offset includes a fixed offset {a1, a2, a3} and a depth-dependent offset {d1, e1, f1}. In this case, by applying the predetermined offset to the first depth pixel, a correction value {x1 * d1 + a1, y1 * e1 + b1, z1 * f1 + c1} of the first depth pixel is obtained.

[0102] Typically, the predetermined offset only describes an offset along the viewing direction from the depth sensor, for example, an offset {0, 0, c1} or a fixed offset {0, 0, c1} and a depth-dependent offset {0, 0, f1}.

[0103] By correcting the values ​​of all depth pixels obtained from the depth image, the processor 22 can obtain a corrected depth image. The corrected depth image may also be referred to as a color-calibrated depth image, an offset-adjusted depth image, a color-shifted depth image, etc. Then, the corrected values ​​of the depth pixels (e.g., in the form of a corrected depth image) may be used as depth information.

[0104] In step 82, a third pose of the surface of the surgical registration device is determined based on the second pose, the surface model data, and the depth information. The third pose is a third position and orientation of the surgical registration device 2 in the sensor device coordinate system 26. Thus, the third pose describes the relative spatial relationship between the surgical registration device 2 and the sensor device 6. The third pose may also be referred to as a corrected second pose, an improved second pose, an adjusted second pose, a final pose, etc.

[0105] The third pose may be determined by the processor 22 by determining, based on the second pose and based on the surface model data, a surface model for describing the surface of at least the second portion of the surgical registration device 2 in the second pose. The surface model is determined by transforming, based on the determined second pose, the surface described by the surface model data from a model coordinate system to a sensor device coordinate system 26. The surface model describes the surface of at least the second portion of the surgical registration device 2 in the sensor device coordinate system 26.

[0106] In a next step, the surface model may be matched with the depth information to determine a third pose. For example, a surface matching algorithm is used to match a three-dimensional coordinate in the sensor device coordinate system 26 with a surface described by the surface model, wherein the three-dimensional coordinate is described by the depth information. As a result of the matching, the second pose may be changed, corrected, adjusted or rectified to a third pose. Although the second pose is determined based on the image alone, the third pose is further determined based on the depth information. In other words, the third pose is determined based on measurements of two sensors using different sensor principles (i.e., based on measurements of an image sensor such as the camera 18 and a sensor such as the depth sensor 20).

[0107] In an example, a surface model is matched to a portion of the depth information that satisfies at least one criterion selected from the following: does not represent a predetermined surface attribute; does not represent an object other than a surgical registration device; and does not represent a portion of a surgical registration device having a predetermined color. The predetermined surface attribute may be an edge, a convex portion, a concave portion, a planar portion of a surface of the surgical registration device 2, the planar portion being smaller than a predetermined minimum size; a portion of the surface of the surgical registration device 2 that is hidden under another portion of the surgical registration device 2, etc. At least one of the predetermined surface attribute and the predetermined color may be specified in the surface model data, or may be specified in another data type obtained before matching the surface model to the portion of the depth information. The surface model is matched to the portion of the depth information by removing all contents of the depth information except for the portion of the depth information, and then matching the three-dimensional coordinates in the sensor device coordinate system 26 (the three-dimensional coordinates being described by the remaining depth information) to the surface described by the surface model. Alternatively, all contents of the depth information except for the portion of the depth information may be ignored for matching.

[0108] As mentioned above, the measurements of the depth sensor are often affected by the color of the detected object. Depending on the color of the detected object, it may be determined to be closer or farther away from the depth sensor 20 than it actually is. In other words, because the depth sensor 20 is sensitive to color, the depth pixels may not describe the correct three-dimensional coordinates. This effect can be used to improve the accuracy of the determined third pose.

[0109] Specifically, determining the third posture may include obtaining second position information indicating a position of the at least one colored region relative to the surface of the surgical registration device 2. In this case, the second position information also includes an indication of a color value of the at least one colored region.

[0110] Then, determining the third posture may also include the step of determining an approximate surface model based on the second posture, the surface model data and the second position information. The approximate surface model describes an estimate of the surface of at least the second part of the surgical registration device 2 in the second posture as described by the depth information. The approximate surface model specifically describes an estimate of the surface of at least the second part of the surgical registration device 2 in the second posture as described by the depth pixel value. The approximate surface model is determined by converting the surface described by the surface model data from the model coordinate system to the sensor device coordinate system 26 based on the second posture. The second position information is used to identify the position of the colored area on the surface of the surface model in the sensor device coordinate system 26 and its respective color value. Subsequently, a predetermined offset is applied to at least one colored area according to its respective color value. As described above, the predetermined offset can be obtained by the processor 22 as part of the surface model data, the second position information or as separate data.

[0111] The surface model is deformed by applying a predetermined offset to at least one colored area. Specifically, the surface area where at least one colored area described by the surface model is located is moved in a specific direction by a specific amount as defined by the applied offset. Therefore, the surface model is changed so that part or all of the surface area described by the surface model where at least one colored area is located is offset compared to the surface of the surface model according to the color value and the predetermined offset associated therewith. In other words, an approximate surface model is generated based on the recognition that the depth sensor is affected by the color value. The approximate surface model is an estimate of the surface of the surgical registration device 2 as it will be detected by the depth sensor 20. The approximate surface model is determined so that it matches the value of the depth pixel better than the surface model. Therefore, the approximate surface model may also be referred to as a color-corrected surface model, a color-adjusted surface model, a depth pixel value estimation model, etc.

[0112] In another step, the approximate surface model can be matched to the depth information to determine the third pose. For example, a surface matching algorithm is used to match the three-dimensional coordinates in the sensor device coordinate system 26 to the surface described by the approximate surface model, wherein the three-dimensional coordinates are described by the depth information. The approximate surface model is preferably matched to the depth pixel values. In other words, the approximate surface model is preferably matched to the uncorrected raw data or measured depth values ​​included in the depth information. On the other hand, the surface model is preferably matched to the corrected values ​​of the depth pixels. In other words, the surface model is preferably matched to the corrected, adjusted or color-calibrated depth values ​​included in the depth information, for example, to a color-corrected depth image.

[0113] As described above, the third posture is also determined based on other information. Specifically, the third posture is determined based on the second posture, the surface model data and the depth information. The third posture can also be determined based on the matching of the surface model with the depth information and / or based on the matching of the approximate surface model with the depth information. One or more initial third postures are determined based on part of the above information, and then corrected based on the remaining information, or the third posture is immediately determined based on all the above information used to determine the third posture.

[0114] The method may further comprise the step of obtaining a known relative spatial relationship between the surgical registration device 2 and the tracking system 4 of the surgical navigation system, specifically a posture of the surgical registration device 2 in the tracking coordinate system 16 .

[0115] As another step, the method may include determining a relative spatial relationship between the sensor device and the tracking system based on the third posture and the known relative spatial relationship. In other words, the processor 22 may determine a transformation between the sensor device coordinate system 26 and the tracking coordinate system 16 based on the determined third posture and the known posture of the surgical registration device 2 in the tracking coordinate system 16.

[0116] Fig.11 Shows Fig.10 Although different reference numerals are used in the two figures, reference has been made to Fig.10 Describes Fig.11Most of the steps shown. In step 1100, the sensor device 6 acquires a raw depth image. In step 1102, temperature calibration is performed based on the raw depth image to obtain a depth image including depth pixels. Temperature calibration removes temperature-based errors from the raw depth image, which errors originate from temperature-related measurements of the depth sensor. In step 1106, the sensor device 6 further acquires a raw color image. In step 1108, distortion calibration is performed to obtain an image, in this case a color image. Distortion calibration removes distortion-based errors from the raw color image, which errors are caused by distortion of optical components or sensor errors of the image sensor used to acquire the raw color image. In step 1110, at least one optical pattern 30 is detected in the color image, which is a QR code in the example shown. Based on the surface model data, the first position information and the detected at least one optical pattern 30, a pose estimate is determined. The pose estimate typically includes an orientation error and a depth error. In order to improve the pose estimate, a color code-based optimization can be performed in step 1112. In this step, at least one colored area is detected and a second position information is obtained. Then, based on the detected at least one colored region and based on the second position information, the pose estimate is improved to determine an improved pose estimate. The improved pose estimate may include a depth error.

[0117] In the case where step 1112 is omitted, the posture estimate corresponds to the second posture. In the case where step 1112 is performed, the improved posture estimate corresponds to the second posture. In the case where the improved posture estimate corresponds to the second posture, the posture estimate may correspond to the initial second posture. Of course, steps 1110 and 1112 may be performed in combination by directly determining the improved posture as the second posture without determining the posture estimate.

[0118] In step 1114, image pixels of the color image are registered with depth pixels of the depth image during inter-camera registration to determine a registered depth image. The registered depth image may include depth pixels and information describing the correlation of each depth pixel with the image pixel. In step 1116, correction values ​​of the depth pixels are determined to obtain a corrected depth image. Using the information describing the correlation of each depth pixel with the image pixel, the correction values ​​are determined based on the values ​​of the depth pixels included in the registered depth image and based on the color values ​​of the pixels of the color image. Alternatively, the registered depth image also includes the color values ​​of the registered image pixels.

[0119] In step 1118, the surface model determined based on the surface model data and the second pose is matched with the corrected depth image by a surface matching algorithm. Please note that the surface matching in step 1118 can be performed based on the second pose, the surface model data and the depth image, for example, in the case where one or both of steps 1114 and 1116 are omitted. As a result of the surface matching, a corrected pose estimate is obtained, which should no longer include depth errors. In one variant, only steps 1110 and 1118 are performed. In this case, the depth image and the pose estimate can be used for the surface matching in step 1118 instead of the corrected depth image and the improved pose estimate.

[0120] In step 1120, a color shift-based optimization is performed. Specifically, the values ​​of the depth pixels are used to determine an approximate surface model and match it to the depth information. It can be seen that the approximate surface model can be determined after determining the registered depth image or directly after determining the depth image. As a result of step 1120, the exact posture of the sensor device 6 relative to the surgical registration device 2 is determined. In one variant, only steps 1110 and 1120 are performed. In this case, the approximate surface model and the posture estimate can be used for the color shift-based optimization in step 1120 instead of the approximate surface model and the corrected posture estimate.

[0121] In case the method ends after step 1118, the corrected pose estimate corresponds to the third pose. In case the method ends after step 1120, the precise pose of the sensor device 6 relative to the surgical registration device 2 corresponds to the third pose. Of course, the determination of the corrected pose estimate can be omitted by performing the surface matching step 1118 and the color shift-based optimization step 1120.

[0122] from Fig.11 It is apparent from the flowchart that some steps may be omitted and / or the order of some steps may be changed. For example, step 1112 may be performed before step 1110. One or more of step 1112, step 1114, and step 1116 may be omitted. Alternatively or additionally, one of step 1118 and step 1120 may be omitted.

[0123] As mentioned above, the methods described herein may be performed by the processor 22. The present disclosure also relates to a computer program comprising instructions which, when executed by a processor such as the processor 22, cause the processor to perform the methods described herein, for example with reference to Fig.10 and Fig.11 Furthermore, the present disclosure relates to a computer readable medium having a computer program stored thereon, and to a data carrier signal carrying the computer program.

[0124] The surgical registration device 2 described herein is formed so that its posture can be determined with high accuracy by the sensor device 6. The surgical registration device 2 described herein is formed so that multiple surface features, such as edges, planar parts, spherical parts, etc., can be detected by the depth sensor 20 of the sensor device 6, such as an augmented reality device, augmented reality glasses, etc. At the same time, the surgical registration device 2 includes at least one optical pattern 30. By using optical detection of at least one optical pattern 30 and also using spatial detection of the surface of the surgical registration device 2 by the depth sensor 20, the position and orientation of the surgical registration device 2 can be accurately determined as a third posture. In other words, two different tracking types (i.e., optical tracking and depth-based tracking) are used to determine the third posture. This improves the accuracy of the determined third posture.

[0125] Additionally, the surgical registration device 2 may include at least one colored area. Thus, the third posture may also be improved by taking into account the position of the detected at least one colored area.

[0126] In addition, the color dependency of the depth pixel values ​​can be used to further improve the determined third pose. According to one method, the depth image can be corrected based on the color values ​​of the pixels registered with the depth pixels of the depth image. The corrected depth image can then be matched to the surface model, thereby improving the matching result and increasing the accuracy of the third pose. According to another method, the color dependency of the depth pixel values ​​is compensated by determining an approximate surface model and matching the approximate surface model to the depth image.

[0127] The features described with respect to the exemplary embodiments shown in the drawings can be easily combined to produce different embodiments. Therefore, it is obvious that the present disclosure can be changed in many ways. Such changes should not be regarded as departing from the scope of the present invention defined by the attached claims.

Claims

1. A computer-implemented method for determining a posture of a surgical registration device, wherein: The surgical registration device comprises a surface and at least one optical pattern arranged on the surface, the method comprising: obtaining an image of at least a first portion of the surgical registration device in a first pose; detecting the at least one optical pattern in the image; obtaining surface model data describing a surface of at least a second portion of the surgical registration device; obtaining first position information indicating a position of the at least one optical pattern relative to a surface of the surgical registration device; determining a second pose of the surface of the surgical registration device based on the detected at least one optical pattern, the surface model data, and the first position information; obtaining depth information describing three-dimensional coordinates of a surface of at least a second portion of the surgical registration device in the first pose; determining a third pose of the surface of the surgical registration device based on the second pose, the surface model data, and the depth information; wherein the second posture and the third posture respectively describe a relative spatial relationship between the surgical registration device and a sensor device including at least one sensor, wherein the at least one sensor is configured to obtain at least one data selected from the depth information and the image, and Wherein, the method further comprises: obtaining a known relative spatial relationship between the surgical registration device and a tracking system of a surgical navigation system; and Based on the third pose and the known relative spatial relationship, a relative spatial relationship between the sensor device and the tracking system is determined.

2. The method according to claim 1, wherein: Determining the third posture includes: determining, based on the second pose and based on the surface model data, a surface model describing a surface of at least a second portion of the surgical registration device in the second pose; and The surface model is matched to the depth information to determine the third pose.

3. The method according to claim 2, wherein: The surface model matches a portion of the depth information, the portion of the depth information satisfying at least one criterion selected from: It does not represent predetermined surface properties; It does not represent objects other than the surgical registration device described; as well as Portions of the surgical registration device having a predetermined color are not represented.

4. The method according to claim 3, wherein: The predetermined surface property is at least one of: an edge of a surface of the surgical registration device; a convex portion of a surface of the surgical registration device; a concave portion of a surface of the surgical registration device; A planar portion of a surface of the surgical registration device that is smaller than a predetermined minimum dimension; and A portion of a surface of the surgical registration device, the portion being hidden beneath another portion of the surgical registration device.

5. The method according to claim 1, wherein: Determining the second posture includes extracting information from the detected at least one optical pattern, the information being indicative of at least one parameter selected from: a size of the at least one optical pattern, the shape of the at least one optical pattern, identification data ID of the at least one optical pattern; a type of the at least one optical pattern, the position of the at least one optical pattern relative to the surface of the surgical registration device, The type of surgical registration device, the shape of the surface of the surgical registration device, the shape of the surface of the second portion of the surgical registration device, and A known relative spatial relationship between the surgical registration device and a tracking system of a surgical navigation system.

6. The method according to claim 1, wherein: The surgical registration device includes at least one colored region disposed on a surface of the surgical registration device, wherein the at least one colored region is different from the at least one optical pattern, and wherein determining the second posture includes: detecting the at least one colored region in the image; obtaining second position information indicating a position of the at least one colored region relative to a surface of the surgical registration device; and The second gesture is also determined based on the detected at least one colored area and the second position information.

7. The method according to claim 1, wherein: Obtaining the depth information includes: Obtaining a plurality of depth pixels; obtaining a plurality of image pixels from the image; registering the plurality of image pixels with the plurality of depth pixels; and Based on the color values ​​of the registered image pixels and based on the values ​​of the registered depth pixels, corrected values ​​of the depth pixels are determined to determine the depth information.

8. The method according to claim 1, wherein: The surgical registration device comprises at least one colored region disposed on a surface of the surgical registration device, wherein determining the third posture comprises: obtaining second position information indicating a position of the at least one colored region relative to a surface of the surgical registration device; determining, based on the second pose, the surface model data and the second position information, an approximate surface model describing an estimate of a surface of at least a second portion of the surgical registration device described by the depth information in the second pose; and The approximate surface model is matched to the depth information to determine the third pose.

9. A computer program product comprising instructions which, when executed by a processor, cause the processor to: obtaining an image of at least a first portion of the surgical registration device in a first pose; detecting the at least one optical pattern in the image; obtaining surface model data describing a surface of at least a second portion of the surgical registration device; obtaining first position information indicating a position of the at least one optical pattern relative to a surface of the surgical registration device; determining a second pose of the surface of the surgical registration device based on the detected at least one optical pattern, the surface model data, and the first position information; obtaining depth information describing three-dimensional coordinates of a surface of at least a second portion of the surgical registration device in the first pose; determining a third pose of the surface of the surgical registration device based on the second pose, the surface model data and the depth information, wherein the second posture and the third posture respectively describe a relative spatial relationship between the surgical registration device and a sensor device including at least one sensor, wherein the at least one sensor is configured to obtain at least one data selected from the depth information and the image, Wherein, when the instructions are executed by a processor, the processor further causes the processor to: obtaining a known relative spatial relationship between the surgical registration device and a tracking system of a surgical navigation system; and Based on the third pose and the known relative spatial relationship, a relative spatial relationship between the sensor device and the tracking system is determined.

10. A sensor device comprising: an image sensor configured to acquire an image; a sensor configured to acquire at least one of depth information and depth pixels; as well as A processor configured to execute the method according to any one of claims 1 to 8.

Images