Glasses and coordinate conversion method for image and depth camera coordinate transformation

Abstract

The application discloses a kind of glasses and coordinate conversion method for image and depth camera coordinate transformation, including frame, the frame includes mirror frame and mirror foot, and the mirror frame is equipped with the positioning mark for magnetic resonance imaging equipment and depth camera identification.The positioning mark in the glasses of the application can be identified by magnetic resonance imaging equipment and depth camera, and the positioning mark identified by depth camera is matched to the positioning mark scanned under magnetic resonance imaging to be one-to-one correspondence, the spatial coordinates of positioning mark under the coordinate system of depth camera are obtained by finding positioning mark under depth camera, and the spatial position of the magnetic resonance image of patient's head under the coordinate system is found.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, specifically to a pair of glasses and a coordinate transformation method for image and depth camera coordinate transformation. Background Technology

[0002] Transcranial magnetic stimulation (TMS) is a commonly used medical treatment for mental illnesses. Magnetic signals can penetrate the skull without attenuation to stimulate brain nerves, and it is painless and non-invasive. However, a persistent challenge in the practical application of TMS is how doctors can accurately pinpoint the stimulation site. Therefore, a TMS positioning and navigation system is needed to address this issue, making it crucial to determine the spatial location of the patient's head within the system. Summary of the Invention

[0003] To address the shortcomings of the prior art, this invention provides glasses and a coordinate transformation method for image and depth camera coordinate transformation, which constructs the spatial positional relationship of the glasses under magnetic resonance imaging and depth camera imaging to achieve position matching.

[0004] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0005] A pair of glasses for image and depth camera coordinate transformation includes a frame, the frame including a lens and temples, and the lens having a positioning mark for identification by a magnetic resonance imaging device and a depth camera.

[0006] Furthermore, there are four positioning markers distributed along the frame, all located on the same plane, and the distance between any two positioning markers is different.

[0007] Furthermore, the positioning mark is hollow spherical, the inner cavity of the positioning mark is filled with vegetable oil, and the outer surface of the positioning mark is coated with reflective material.

[0008] Furthermore, the outer diameter of the positioning mark is 9.5 mm and the inner diameter is 8 mm.

[0009] Furthermore, the positioning markers include two types: reflective spheres and spheres filled with vegetable oil. The reflective spheres and spheres filled with vegetable oil have the same outer diameter and are fixed on two frames with the same shape. The relative positions of the reflective spheres and spheres filled with vegetable oil on the frames are consistent.

[0010] Furthermore, a second positioning mark is set on each of the two temples.

[0011] A coordinate transformation method for glasses used for image and depth camera coordinate transformation, comprising the following steps:

[0012] S1. By capturing real-time images from the depth camera, the coordinates of four positioning markers in the depth camera coordinate system are obtained, which we represent as A(X0, Y0, Z0), B(X1, Y1, Z1), C(X2, Y2, Z2), and D(X3, Y3, Z3). The state matrix of the glasses can be shown in the figure below:

[0013]

[0014] S2. Given that the coordinates of point A in the frame's own coordinate system are (0, 0, 0), the translation matrix can be obtained.

[0015]

[0016] Using the translation matrix, the matrix constructed by transforming the coordinates of the entire glasses in the depth camera coordinate system to the world coordinate system is as follows:

[0017]

[0018] S3. Let the rotation matrix be:

[0019]

[0020] The formula for calculating the transformation of the coordinate matrix is:

[0021]

[0022] Multiply both sides of this formula by the inverse of the middle matrix to calculate the specific value of the rotation matrix.

[0023] Based on the specific values ​​of the rotation matrix, the rotation-translation matrix is ​​constructed as follows:

[0024]

[0025] S4. Take the coordinates of a point in the magnetic resonance image in the frame coordinate system. Assuming the coordinates are (X, Y, Z), calculate the coordinates of that point in the depth camera coordinate system using the following formula.

[0026]

[0027] That is, (X',Y',Z') are the coordinates of the point (X,Y,Z) in the magnetic resonance image under the depth camera.

[0028] Furthermore, coordinate calibration is required before coordinate transformation. The calibration steps are as follows:

[0029] T1: Marker; Four positioning markers are located on the same plane, and the coordinates of the positioning markers under magnetic resonance imaging are marked;

[0030] T2: Select any three positioning markers as a plane, calculate the distance from the other positioning marker to this plane; determine the error situation. If the distance is 0, it means that the positions of the four positioning markers in the image have no error and no calibration is required; if the distance is not 0, proceed to the next step.

[0031] T3: Iterate and repeat T2 to find the case with the largest distance; mark the midpoint of the straight line connecting the positioning marker with the corresponding plane projection point when the distance is the largest, and update the coordinates of the positioning marker to the coordinates of the midpoint as the first correction;

[0032] T4. Repeat steps T2 and T3 until the error falls within the preset error range to complete the calibration.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] This invention identifies a scanning positioning marker using a magnetic resonance scanner, determines the relative spatial position of the positioning marker and the patient's head in the magnetic resonance image, and then identifies the positioning marker under a depth camera. The positioning marker is matched one-to-one with the signal generated by the positioning marker scanned in the magnetic resonance image. By finding the positioning marker under the depth camera and obtaining the spatial coordinates of the positioning marker in the depth camera coordinate system, the spatial position of the patient's head in the magnetic resonance image in that coordinate system can be found. The structure is simple and the alignment is accurate.

[0035] The coordinate transformation method provided by this invention has low computational cost, fast response, and accurate results; coordinate calibration before coordinate transformation can further improve accuracy. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is an overall view of an embodiment of the invention;

[0038] Figure 2 yes Figure 1 The front view;

[0039] Figure 3 yes Figure 1 Top view.

[0040] Attached reference numerals: 1-frame, 2-temples, 3-positioning mark, 4-secondary positioning mark. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0042] A type of eyewear for image and depth camera coordinate transformation, such as Figures 1-3 As shown, the glasses include a frame, which includes a frame 1 and temples 2. The frame 1 is provided with a positioning mark 3 for identification by magnetic resonance imaging equipment and / or depth camera.

[0043] There are four positioning markers 3 in total, distributed along the frame 1. The positioning markers 3 are located on the same plane, and the distance between any two positioning markers 3 is different.

[0044] In one embodiment, the positioning mark 3 is a hollow sphere, the inner cavity of the positioning mark 3 is filled with vegetable oil, and the outer surface of the positioning mark is coated with a reflective material, such as reflective glass microspheres; preferably, the outer diameter of the positioning mark is 9.5 mm and the inner diameter is 8 mm; the vegetable oil in the inner cavity of the positioning mark can be identified by a magnetic resonance scanner, and the reflective material on the surface can be identified by a depth camera.

[0045] In another preferred embodiment, the positioning marker includes two types: a reflective ball and a ball filled with vegetable oil. The reflective ball and the ball filled with vegetable oil have the same outer diameter and are fixed on two frames with the same shape. The relative positions of the reflective ball and the ball filled with vegetable oil on the frames are consistent. The frame filled with vegetable oil is used for magnetic resonance imaging, and the frame with the reflective ball is used for depth camera imaging.

[0046] The positioning markers identified by the depth camera are matched one-to-one with the positioning markers scanned under the magnetic resonance imaging. By finding the positioning markers under the depth camera and obtaining the spatial coordinates of the positioning markers in the depth camera coordinate system, the spatial position of the patient's head magnetic resonance image in that coordinate system can be found.

[0047] Specifically, the coordinate transformation method for the glasses used for image and depth camera coordinate transformation is as follows:

[0048] S1. By acquiring real-time images from the depth camera, we first obtain the coordinates of four positioning markers in the depth camera coordinate system using the camera's algorithm. We use A(X0, Y0, Z0), B(X1, Y1, Z1), C(X2, Y2, Z2), and D(X3, Y3, Z3) to represent these coordinates. The state matrix of the glasses can be shown in the following figure:

[0049]

[0050] S2. Since the coordinates of point A in the coordinate system of the mirror frame are (0, 0, 0), the translation matrix can be obtained.

[0051]

[0052] Using the translation matrix, the matrix constructed by transforming the coordinates of the entire glasses in the depth camera coordinate system to the world coordinate system is as follows:

[0053]

[0054] S3. Let the rotation matrix be:

[0055]

[0056] We know that the formula for transforming a coordinate matrix is:

[0057]

[0058] Multiply both sides of this formula by the inverse of the middle matrix to calculate the specific value of the rotation matrix.

[0059] Now that we know the specific values ​​of the rotation matrix, we can construct a rotation-translation matrix by combining the rotation and translation matrices as follows:

[0060]

[0061] S4. Obtain the coordinates of a point in the magnetic resonance image in the lens frame coordinate system. These coordinates (relative to the lens frame coordinate system) were already obtained during the scanning process using the glasses. Assuming the coordinates are (X, Y, Z), the coordinates of this point in the depth camera coordinate system can be calculated using the following formula.

[0062]

[0063] That is, (X', Y', Z') are the coordinates of the point (X, Y, Z) in the magnetic resonance image under the depth camera.

[0064] Because of the different body positions of people during MRI, and the soft padding used to fix the head during the scan may also deform the eyeglass frame, the spatial position relationship of the four positioning marks on the eyeglass frame scanned under MRI may be slightly misaligned compared to the actual spatial position relationship of the four positioning marks on the eyeglass frame.

[0065] To solve this problem, coordinate calibration is required before coordinate transformation. The specific steps are as follows:

[0066] T1: Marker; Four positioning markers are located on the same plane. The coordinates of the positioning markers under magnetic resonance imaging are marked as points M, N, P, and Q respectively.

[0067] T2: Select any three positioning markers as a plane and calculate the distance from the other positioning marker to this plane; for example, first select points M, N, and P as a plane, and then calculate the distance from point Q to this plane; determine the error situation. If the distance is 0, it means that the positions of the four positioning markers in the image have no error and no calibration is required; if the distance is not 0, proceed to the next step.

[0068] T3: Iterate and repeat T2 to find the case with the largest distance; mark the midpoint of the straight line connecting the positioning marker and the corresponding plane projection point when the distance is the largest, and update the coordinates of the positioning marker to the coordinates of the midpoint as the first correction;

[0069] T4. Repeat steps T2 and T3 until the error falls within the preset error range to complete the calibration.

[0070] By using a transformation matrix, the coordinates of the four positioning markers under the depth camera and the four positioning markers under magnetic resonance imaging are transformed accordingly.

[0071] In addition to the misalignment of positioning marks on the frame during MRI, there may also be misalignment of the entire frame plane. To address this, the present invention also provides a second positioning mark 4 on each of the two temples 2, used to correct the coordinates of the positioning marks under MRI when the patient is lying on their side, and to calculate the coordinates of the human brain under the depth camera through a series of operations.

[0072] The specific correction method is as follows:

[0073] The coordinates of the two second positioning marks on the temples are set as (XR, YR, ZR) and (XL, YL, ZL) respectively in the eyeglasses' own coordinate system.

[0074] The coordinates of the two second positioning marks on the temple in the magnetic resonance coordinate system are set as (XR0, YR0, ZR0) and (XL0, YL0, ZL0), respectively.

[0075] The corrected coordinates of the second positioning marker in the magnetic resonance coordinate system are represented by the following matrix:

[0076]

[0077] The coordinates of the second positioning marker detected by the camera in the depth camera coordinate system are represented by the following matrix:

[0078]

[0079] The shear transformation matrix T1 for the four positioning markers on the aforementioned frame plane, from the magnetic resonance image to the coordinates under the depth camera, is calculated as follows:

[0080]

[0081] Multiply the matrix on the right by the transpose of the second positioning mark on the telescope's foot in the magnetic resonance imaging coordinates:

[0082]

[0083] The result It is the coordinates of the second positioning marker under the depth camera after offset (in the squeezed state). However, because the glasses are in a normal state under the depth camera, they will be inconsistent with the spatial coordinates detected by the depth camera. At this time, the shear transformation matrix used to correct the human brain coordinates can be calculated.

[0084] The coordinates of the second positioning marker on the front of the telescope, detected by the depth camera, are:

[0085]

[0086] Then assume that the rotation and translation matrix T2 is...

[0087]

[0088] Solving the translation matrix

[0089]

[0090] Multiply both sides on the far right The inverse matrix is ​​obtained. The specific value.

[0091] The coordinates of a point on the human brain under magnetic resonance imaging (MRI) are (X,Y,Z). When these coordinates are multiplied by the previous transformation matrix T1, the resulting image of the brain's coordinates under the depth camera is the offset MRI image, denoted as (X',Y',Z'). Now, we need to multiply this by a correction matrix T2 to obtain (X”,Y”,Z”), which is the true coordinate value of this location under the depth camera.

[0092] When a patient lies on their side on an MRI bed wearing the aforementioned glasses, factors such as the coils and the patient's position may exert pressure on the glasses, easily causing them to deform. This can lead to misalignment of some positioning marks on the frame compared to when glasses are worn normally. However, in this case, the two secondary positioning marks on the temples generally correspond to their positions on the brain when glasses are worn normally.

[0093] Since the four positioning marks on the frame are on a plane, they will not change under normal circumstances. However, the temples will tilt due to pressure, causing the positioning marks on the frame to be misaligned from their original positions. That is, the straight line formed by the connection between the temples and the frame will no longer be perpendicular to the plane formed by the four positioning marks on the frame, but will form an angle. This also means that the human brain coordinates calculated from the coordinates of the four positioning marks on the frame will have a large error.

[0094] Using the shear transformation in the affine transformation described above, a shear transformation matrix is ​​calculated using the offset second positioning identifier and the frame plane positioning identifier. Before coordinate correction, the coordinates of the human brain under the depth camera are directly calculated from the coordinates of the glasses under the depth camera. Without any processing, this is offset from the actual target; that is, the actual coordinates are to the left or right of the calculated coordinates. However, this coordinate can be corrected using the previously calculated shear transformation matrix, and the corrected coordinates are the true coordinates of the human brain under the depth camera.

[0095] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A coordinate transformation method for glasses used for image and depth camera coordinate transformation, characterized in that: The system includes a frame, which includes a frame (1) and temples (2). The frame (1) is provided with positioning marks (3) for magnetic resonance imaging equipment and depth camera to identify. There are four positioning marks (3) in total, distributed along the frame (1). The positioning marks (3) are located on the same plane, and the distance between any two positioning marks (3) is different. A second positioning mark (4) is provided on each of the two temples (2). When in use, the patient wears glasses and matches the positioning marks identified by the depth camera with the positioning marks scanned by the magnetic resonance imaging equipment. By finding the positioning marks under the depth camera, the spatial coordinates of the positioning marks in the depth camera coordinate system are obtained. According to the corresponding transformation relationship of the positioning marks, the spatial position of the magnetic resonance image of the patient's head in the depth camera coordinate system is obtained. A shear transformation matrix is ​​calculated using the offset second positioning marks and the positioning marks on the frame plane, which is used to correct the coordinates of the positioning marks under the magnetic resonance imaging when the patient is lying on his side. The steps of the coordinate transformation method are as follows: S1. By acquiring real-time images from the depth camera, the coordinates of four positioning markers in the depth camera coordinate system are obtained, which we represent as A(X0, Y0, Z0), B(X1, Y1, Z1), C(X2, Y2, Z2), and D(X3, Y3, Z3). The state matrix of the glasses is represented as follows: S2. Given that the coordinates of point A in the coordinate system of the mirror frame are (0, 0, 0), obtain the translation matrix. Using the translation matrix, the matrix constructed by transforming the coordinates of the entire glasses in the depth camera coordinate system to the world coordinate system is as follows: S3. Let the rotation matrix be: The formula for calculating the transformation of the coordinate matrix is: = Multiply both sides of this formula by the inverse of the middle matrix to calculate the specific value of the rotation matrix. Based on the specific values ​​of the rotation matrix, the rotation-translation matrix is ​​constructed as follows: S4. Take the coordinates of a point in the magnetic resonance image in the frame coordinate system. Assuming the coordinates are (X, Y, Z), calculate the coordinates of that point in the depth camera coordinate system using the following formula. That is (X) ’ ,Y ’ Z ’ ) is the coordinate of the point (X, Y, Z) in the magnetic resonance image as seen by the depth camera.

2. The coordinate transformation method for eyeglasses used for image and depth camera coordinate transformation according to claim 1, characterized in that: The positioning mark (3) is a hollow sphere, and the inner cavity of the positioning mark (3) is filled with vegetable oil. The outer surface of the positioning mark is coated with reflective material.

3. The coordinate transformation method for eyeglasses used for image and depth camera coordinate transformation according to claim 2, characterized in that: The outer diameter of the positioning mark is 9.5 mm and the inner diameter is 8 mm.

4. The coordinate transformation method for eyeglasses used for image and depth camera coordinate transformation according to claim 1, characterized in that: The positioning markers include two types: reflective spheres and spheres filled with vegetable oil. The reflective spheres and spheres filled with vegetable oil have the same outer diameter and are fixed on two frames with the same shape. The relative positions of the reflective spheres and spheres filled with vegetable oil on the frames are consistent.

5. The coordinate transformation method for eyeglasses used for image and depth camera coordinate transformation according to claim 1, characterized in that: Before coordinate transformation, coordinate calibration is required. The calibration steps are as follows: T1: Marker; Four positioning markers are located on the same plane, and the coordinates of the positioning markers under magnetic resonance imaging are marked; T2: Select any three positioning markers as a plane, calculate the distance from the other positioning marker to this plane; determine the error situation. If the distance is 0, it means that the positions of the four positioning markers in the image have no error and no calibration is required; if the distance is not 0, proceed to the next step. T3: Iterate and repeat T2 to find the case with the largest distance; mark the midpoint of the straight line connecting the positioning marker with the corresponding plane projection point when the distance is the largest, and update the coordinates of the positioning marker to the coordinates of the midpoint as the first correction; T4. Repeat steps T2 and T3 until the error falls within the preset error range to complete the calibration.

Images