Multi-angle projection surgical navigation device based on near-infrared imaging

Abstract

The invention discloses a multi-angle projection surgical navigation device based on near-infrared imaging, which relates to the technical field of fluorescent surgical navigation and comprises a calculation module, an infrared camera and a plurality of projectors, the infrared camera is arranged in the middle, the optical axis is perpendicular to the imaging area, and invisible fluorescence signals of the imaging area are collected and converted into image data; the plurality of projectors are uniformly distributed around the infrared camera, the optical axes all point to the center of an imaging area, and the distances between the projectors and the infrared camera are equal; the calculation module calculates image data which should be projected by the projectors through a mathematical method by establishing a mathematical coordinate conversion model between the infrared camera and the projectors, and real-time fluorescence images collected by the infrared camera are accurately projected into the imaging view of the camera from different angles in the form of visible light. The defects that in an existing fluorescent surgical navigation technology, sight needs to be transferred during image observation, and the projection position is not accurate are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of fluorescent surgical navigation technology, more particularly to a multi-angle projection surgical navigation device based on near-infrared imaging. BACKGROUND

[0002] In the field of fluorescent surgical navigation, traditional technology usually relies on an infrared camera to collect fluorescent signals (invisible light) of a surgical area and transmits the collected images to a separate display device (such as a display, monitor) for a doctor to observe. The doctor needs to frequently shift his line of sight between the surgical operation area and the display device during the operation, which not only easily causes the interruption of operation continuity, but also may cause the deviation of the judgment of the spatial position of the surgical area due to the shift of the line of sight, affecting the accuracy of the operation.

[0003] Some existing technologies attempt to project the collected images to the surgical area, but there is a problem of low projection position accuracy. The main reason is that the accurate coordinate mapping relationship between the infrared camera and the projector is not established, and the projection image data of each projector cannot be accurately calculated according to the camera imaging field of view. In addition, when multiple projectors work cooperatively, the matching degree of the projection range of each projector and the camera field of view is insufficient, which easily causes distortion in the overlapping area of the projection or a projection blind area, further reducing the navigation reliability.

[0004] Therefore, how to overcome the defects of image observation requiring shifting the line of sight and inaccurate projection position in the existing fluorescent surgical navigation technology is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a multi-angle projection surgical navigation device based on near-infrared imaging, which establishes a mathematical coordinate conversion model between an infrared camera and multiple projectors to calculate the image data that each projector should project by mathematical method, realizes the accurate projection of real-time fluorescent images collected by the infrared camera in the form of visible light from different angles within the camera imaging field of view, and improves the convenience and accuracy of fluorescent surgical navigation.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions: A multi-angle projection surgical navigation device based on near-infrared imaging, comprising: a calculation module, one infrared camera and a plurality of projectors; the infrared camera is centrally arranged, the optical axis is perpendicular to the imaging area, the invisible fluorescent signal of the imaging area is collected and converted into image data; the plurality of projectors are uniformly distributed around the infrared camera, the optical axes of the projectors all point to the center of the imaging area, and the distances between the projectors and the infrared camera are equal; the calculation module performs the following steps: Predefine the world coordinate system, camera coordinate system, projector coordinate system, camera imaging plane coordinate system, and projector imaging plane coordinate system, and determine the parameters and intrinsic parameter matrices of each coordinate system; Establish the coordinate transformation relationships from the world coordinate system to the camera coordinate system, from the camera coordinate system to the camera imaging plane coordinate system, from the world coordinate system to the projector coordinate system, and from the projector coordinate system to the projector imaging plane coordinate system; According to the coordinate transformation relationship, the pixel coordinates of the image captured by the infrared camera are transformed into world coordinates, and then the world coordinates are transformed into pixel coordinates of the imaging plane coordinate system of each projector. The grayscale value of the camera image is assigned to the corresponding pixel of the imaging plane of the projector to obtain the projected image data of each projector.

[0007] Preferably, the world coordinate system has its origin at the center of the imaging area, with the X and Y axes lying in the plane of the imaging area, and the Z axis perpendicular to the imaging area and pointing upwards; the camera coordinate system has its origin at the optical center C of the infrared camera, with the Xc and Yc axes parallel to the X and Y axes of the world coordinate system, respectively, and the Zc axis coinciding with the optical axis of the infrared camera; the distance from the optical center of the camera to the origin of the world coordinate system is the camera height H; the coordinates of the optical center of the i-th projector in the projector coordinate system are (L... sinθ cosα_i, L sinθ sinα_i, L cosθ), where L is the distance from the optical center of the projector to the origin of the world coordinate system, θ is the angle between the optical axis of the projector and the Z-axis of the world coordinate system, and α_i=2π(i-1) / N is the angle between the i-th projector and the X-axis of the world coordinate system; the intrinsic parameter matrix of the camera in the camera imaging plane coordinate system is Kc= [[f_c / sx, 0, u0], [0, f_c / sy, v0], [0,0, 1]], where f_c is the focal length of the camera, sx and sy are the pixel sizes of the camera, and (u0, v0) is the center coordinate of the camera imaging plane; the intrinsic parameter matrix of the projector is Ki= [[f_i / sxi, 0, u0i], [0, f_i / syi, v0i], [0, 0, 1]], where f_i is the focal length of the projector, sxi and syi are the pixel sizes of the projector, and (u0, v0) is the center coordinate of the camera imaging plane; the intrinsic parameter matrix of the projector is Ki= [[f_i / sxi, 0, u0i], [0, f_i / syi, v0i], [0, 0, 1]], where f_i is the focal length of the projector, sxi and syi are the pixel sizes of the projector, and (u0i, v0) is the center coordinate of the camera imaging plane. v0i) represents the center coordinates of the projector's imaging plane; the imaging plane of the i-th projector in the projector's imaging plane coordinate system is a two-dimensional plane with the center of the imaging plane as the origin. The Ui axis and Vi axis correspond to the Xi axis and Yi axis of the projector's coordinate system, respectively, with the unit being pixels. The intrinsic parameter matrix of the projector is Ki, Ki = [[f_i / sxi, 0, u0i], [0, f_i / syi, v0i], [0, 0, 1]], which includes the focal length f_i, pixel size sxi, and syi. (u0i, v0i) represents the pixel coordinates of the center of the projector's imaging plane in the Ii coordinate system.

[0008] Preferably, the coordinate transformation relationship from the world coordinate system to the camera coordinate system specifically includes: Let the coordinates of any point P in the imaging area be W=(X,Y,0) in the world coordinate system, and its coordinates in the camera coordinate system be C=(Xc,Yc,Zc). The coordinates of the origin C of the camera coordinate system in the world coordinate system are (0,0,H), and the axes of the two coordinate systems are parallel. The transformation relationship is: Xc =X; Yc = Y; Zc =-H. The negative sign indicates that the optical center of the camera is in the positive direction of the Z-axis in the world coordinate system, and point P is in the negative direction of the Z-axis.

[0009] Preferably, the coordinate transformation relationship from the camera coordinate system to the camera imaging plane coordinate system specifically includes: according to the principle of perspective projection, the pixel coordinates (Ic_u, Ic_v) of point P in the Ic coordinate system of the camera imaging plane satisfy: [Ic_u; Ic_v; 1] = Kc Substituting [Xc; Yc; Zc] into Xc, Yc, and Zc, we get: Ic_u = (f_c / sx) X / (-H) + u0;Ic_v = (f_c / sy) Y / (-H) + v0; The image data acquired by the infrared camera is the gray value G(Ic_u, Ic_v) corresponding to each pixel (Ic_u, Ic_v).

[0010] Preferably, the coordinate transformation relationship from the world coordinate system to the projector coordinate system specifically includes: the coordinates of the optical center Pi of the i-th projector in the world coordinate system are determined according to its distribution angle. Let the angle between the i-th projector and the X-axis be α_i = 2π(i-1) / N, the optical axis of the projector point to point O, the angle between the optical axis and the Z-axis be θ, and the coordinates of Pi in the Z-axis direction be L. cosθ, the distance from the projection of the XY plane to point O is L. sinθ; The transformation between the projector coordinate system and the world coordinate system is achieved through the rotation matrix Ri and the translation vector Ti, [Xi; Yi; Zi] = Ri [X - PwX; Y - PwY; 0 - PwZ]; where (PwX, PwY, PwZ) are the world coordinates of Pi, and Ri is a 3×3 rotation matrix.

[0011] Preferably, the coordinate transformation relationship from the projector coordinate system to the projector imaging plane coordinate system specifically includes: the pixel coordinates (Ii_u, Ii_v) of point P in the Ii coordinate system of the i-th projector imaging plane satisfying: [Ii_u; Ii_v; 1] = Ki [Xi; Yi; Zi].

[0012] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multi-angle projection surgical navigation device based on near-infrared imaging, with the following beneficial effects: 1. Precise projection position: By establishing a mathematical transformation model of multiple coordinate systems, the projection image data of each projector is directly calculated, avoiding the delay and error of software tools, ensuring that the real-time image captured by the infrared camera can be accurately projected onto the imaging area, and the projection error can be controlled within 0.5 pixels.

[0013] 2. Improved surgical convenience: Doctors do not need to shift their gaze to observe the independent monitor and can directly see the fluorescent image projected by visible light in the surgical area, reducing operation interruptions and improving surgical efficiency.

[0014] 3. Multi-angle coverage: Multiple projectors are evenly distributed to achieve full-range projection coverage of the camera's imaging field of view, with no projection blind spots, and the projected images from each projector are seamlessly connected, avoiding distortion in overlapping areas.

[0015] 4. High reliability: The mathematical calculation method involves less computation and the device has strong anti-interference ability in complex surgical environments, reducing the risk of equipment failure. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a structural schematic diagram provided for the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] This invention discloses a multi-angle projection surgical navigation device based on near-infrared imaging. The device includes one infrared camera and N projectors (N≥2). The infrared camera is centrally located, with its optical axis perpendicular to the surgical imaging area (assumed to be the XY plane), and is responsible for acquiring invisible fluorescence signals within the imaging area and converting them into image data. The N projectors are evenly distributed around the infrared camera, and the optical axes of all projectors point to the central origin O of the imaging area. Each projector is equidistant from the infrared camera, and the angle between adjacent projectors is 360° / N.

[0020] The calculation module performs the following steps: S1 predefines the world coordinate system, camera coordinate system, projector coordinate system, camera imaging plane coordinate system, and projector imaging plane coordinate system, and determines the parameters and intrinsic parameter matrices of each coordinate system; S2 establishes the coordinate transformation relationships from the world coordinate system to the camera coordinate system, from the camera coordinate system to the camera imaging plane coordinate system, from the world coordinate system to the projector coordinate system, and from the projector coordinate system to the projector imaging plane coordinate system; S3 transforms the pixel coordinates of the infrared camera image into world coordinates based on the coordinate transformation relationship, then transforms the world coordinates into pixel coordinates of the imaging plane coordinate system of each projector, and assigns the grayscale value of the camera image to the corresponding pixel of the imaging plane of the projector, thus obtaining the projected image data of each projector.

[0021] Step S3 involves transforming the camera imaging plane coordinate system to the camera coordinate system, then from the camera coordinate system to the world coordinate system; from the world coordinate system to the projector coordinate system, and then from the projector coordinate system to the projector imaging plane coordinate system.

[0022] Device structure as followsFigure 1 As shown in the figure, the central position of the infrared camera, the uniform distribution of the N projectors, and the directional relationship between the optical axis of each device and the center of the imaging area are clearly illustrated: World coordinate system (W): with the center O of the imaging area as the origin, the X and Y axes lie in the plane of the imaging area, and the Z axis is perpendicular to the imaging area and points upward (consistent with the optical axis of the infrared camera), with the unit being mm.

[0023] Camera coordinate system (C): The origin is the optical center C of the infrared camera. The Xc axis and Yc axis are parallel to the X-axis and Y-axis of the world coordinate system, respectively. The Zc axis coincides with the optical axis of the infrared camera (i.e., the Zc axis is in the same direction as the Z-axis of the world coordinate system). The distance from the optical center C of the camera to the origin O is H (i.e., the camera height), and the unit is mm.

[0024] Projector coordinate system (Pi): The optical center of the i-th projector (i=1,2,...,N) is Pi, with Pi as the origin. The Xi and Yi axes lie in the imaging plane of the projector, and the Zi axis coincides with the optical axis of the projector (pointing to the origin O). Let L be the distance from the optical center Pi of the projector to the origin O, and let θ be the angle between the optical axis of the projector and the Z axis (determined by the mechanical structure of the device).

[0025] Camera Imaging Plane Coordinate System (Ic): The imaging plane of an infrared camera is a two-dimensional plane, with the center of the imaging plane as the origin. The Uc axis and Vc axis correspond to the Xc axis and Yc axis of the camera coordinate system, respectively, with the unit being pixels. The camera's intrinsic parameter matrix is ​​Kc, which includes the focal length f_c (mm), pixel size sx, and sy (mm / pixel), i.e.: Kc = [[f_c / sx, 0, u0], [0, f_c / sy, v0], [0, 0, 1]] Where (u0, v0) are the coordinates (in pixels) of the center of the imaging plane in the Ic coordinate system.

[0026] Projector imaging plane coordinate system (Ii): The imaging plane of the i-th projector is a two-dimensional plane with the center of the imaging plane as the origin. The Ui axis and Vi axis correspond to the Xi axis and Yi axis of the projector coordinate system, respectively, and the unit is pixels. The intrinsic parameter matrix of the projector is Ki, which includes the focal length f_i (mm), pixel size sxi, syi (mm / pixel), that is: Ki = [[f_i / sxi, 0, u0i], [0, f_i / syi, v0i], [0, 0, 1]] Where (u0i, v0i) are the coordinates (in pixels) of the center of the projector's imaging plane in the Ii coordinate system.

[0027] Transformation from world coordinate system to camera coordinate system: Let the coordinates of any point P within the imaging region be W=(X,Y,0) in the world coordinate system (since the imaging region is the Z=0 plane), and its coordinates in the camera coordinate system be C=(Xc,Yc,Zc). Since the origin C of the camera coordinate system has coordinates (0,0,H) in the world coordinate system, and the axes of the two coordinate systems are parallel, the transformation relationship is as follows: Xc = X Yc = Y Zc = 0 - H = -H (The negative sign indicates that the camera's optical center is in the positive Z-axis direction of the world coordinate system, and point P is in the negative Z-axis direction) Transformation from camera coordinate system to camera imaging plane coordinate system: According to the principle of perspective projection, the pixel coordinates (Ic_u, Ic_v) of point P in the Ic coordinate system of the camera imaging plane satisfy: [Ic_u; Ic_v; 1] = Kc [Xc; Yc; Zc] Substituting Xc, Yc, and Zc, we get: Ic_u = (f_c / sx) X / (-H) + u0 Ic_v = (f_c / sy) Y / (-H) + v0 The image data acquired by the infrared camera is the gray value G(Ic_u, Ic_v) corresponding to each pixel (Ic_u, Ic_v).

[0028] Transformation from world coordinate system to projector coordinate system: The coordinates of the optical center Pi of the i-th projector in the world coordinate system are determined by its distribution angle. Let the angle between the i-th projector and the X-axis be α_i = 2π(i-1) / N, then the world coordinates of Pi are (L... sinθ cosα_i, L sinθ sinα_i, L (cosθ) (Since the projector's optical axis points to point O, and the angle between the optical axis and the Z-axis is θ, the coordinate of Pi in the Z-axis direction is L) cosθ, the distance from the projection of the XY plane to point O is L. sinθ).

[0029] The transformation between the projector coordinate system and the world coordinate system is achieved through the rotation matrix Ri and the translation vector Ti, that is: [Xi; Yi; Zi] = Ri [X - PwX; Y - PwY; 0 - PwZ] Where (PwX, PwY, PwZ) are the world coordinates of Pi, and Ri is a 3×3 rotation matrix to ensure that the Zi axis of the projector points to point O (the origin of the world coordinate system). The specific matrix elements are determined by θ and α_i.

[0030] Transformation from projector coordinate system to projector imaging plane coordinate system: The pixel coordinates (Ii_u, Ii_v) of point P in the Ii coordinate system of the i-th projector imaging plane satisfy: [Ii_u; Ii_v; 1] = Ki [Xi; Yi; Zi] For any pixel (Ic_u, Ic_v) in an image captured by an infrared camera, its corresponding world coordinates (X,Y,0) can be obtained by inverse calculation using the following formula: X = (-H / sx) (Ic_u - u0) / f_c Y = (-H / sy) (Ic_v - v0) / f_c Substituting (X,Y,0) into the transformation formula from the world coordinate system to the projector imaging plane coordinate system, we obtain the pixel coordinates (Ii_u, Ii_v) of this point on the imaging plane of the i-th projector. Assigning the grayscale value G(Ic_u, Ic_v) of the pixel (Ic_u, Ic_v) in the camera image to the pixel (Ii_u, Ii_v) on the projector imaging plane yields the image data Gi(Ii_u, Ii_v) to be projected by the i-th projector. The corresponding projector then projects this image data onto the imaging area in visible light form.

[0031] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0032] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-angle projection surgical navigation device based on near-infrared imaging, characterized in that, include: The system comprises a computing module, an infrared camera, and several projectors. The infrared camera is centrally located with its optical axis perpendicular to the imaging area, acquiring invisible fluorescence signals from the imaging area and converting them into image data. The projectors are evenly distributed around the infrared camera, with their optical axes all pointing towards the center of the imaging area, and each projector is equidistant from the infrared camera. The computing module performs the following steps: Predefine the world coordinate system, camera coordinate system, projector coordinate system, camera imaging plane coordinate system, and projector imaging plane coordinate system, and determine the parameters and intrinsic parameter matrices of each coordinate system; Establish the coordinate transformation relationships from the world coordinate system to the camera coordinate system, from the camera coordinate system to the camera imaging plane coordinate system, from the world coordinate system to the projector coordinate system, and from the projector coordinate system to the projector imaging plane coordinate system; According to the coordinate transformation relationship, the pixel coordinates of the image captured by the infrared camera are transformed into world coordinates, and then the world coordinates are transformed into pixel coordinates of the imaging plane coordinate system of each projector. The grayscale value of the camera image is assigned to the corresponding pixel of the imaging plane of the projector to obtain the projected image data of each projector.

2. The multi-angle projection surgical navigation device based on near-infrared imaging according to claim 1, characterized in that, The world coordinate system has its origin at the center of the imaging area, with the X and Y axes lying in the plane of the imaging area and the Z axis perpendicular to the imaging area and pointing upwards. The camera coordinate system has its origin at the optical center C of the infrared camera, with the Xc and Yc axes parallel to the X and Y axes of the world coordinate system, respectively, and the Zc axis coinciding with the optical axis of the infrared camera. The distance from the optical center of the camera to the origin of the world coordinate system is the camera height H. The coordinates of the optical center of the i-th projector in the world coordinate system are (L... sinθ cosα_i, L sinθ sinα_i, L cosθ), where L is the distance from the optical center of the projector to the origin of the world coordinate system, θ is the angle between the optical axis of the projector and the Z-axis of the world coordinate system, and α_i=2π(i-1) / N is the angle between the i-th projector and the X-axis of the world coordinate system; the intrinsic parameter matrix of the camera in the camera imaging plane coordinate system is Kc= [[f_c / sx, 0, u0], [0, f_c / sy, v0], [0, 0, 1]], where f_c is the focal length of the camera, sx and sy are the pixel sizes of the camera, and (u0, v0) is the center coordinate of the camera imaging plane; the intrinsic parameter matrix of the projector is Ki= [[f_i / sxi, 0, u0i], [0, f_i / syi, v0i], [0, 0, 1]], where f_i is the focal length of the projector, sxi and syi are the pixel sizes of the projector, and (u0i, v0i) is the center coordinate of the projector's imaging plane; the imaging plane of the i-th projector in the projector's imaging plane coordinate system is a two-dimensional plane with the center of the imaging plane as the origin. The Ui axis and Vi axis correspond to the Xi axis and Yi axis of the projector's coordinate system, respectively, with the unit being pixels. The intrinsic parameter matrix of the projector is Ki, Ki = [[f_i / sxi, 0, u0i], [0, f_i / syi, v0i], [0, 0, 1]], which includes the focal length f_i, pixel size sxi, syi, and (u0i, v0i) is the pixel coordinate of the center of the projector's imaging plane in the Ii coordinate system.

3. The multi-angle projection surgical navigation device based on near-infrared imaging according to claim 2, characterized in that, The coordinate transformation relationship from the world coordinate system to the camera coordinate system specifically includes: Let the coordinates of any point P in the imaging region be W=(X,Y,0) in the world coordinate system, and its coordinates in the camera coordinate system be C=(Xc,Yc,Zc). The coordinates of the origin C of the camera coordinate system in the world coordinate system are (0,0,H), and the axes of the two coordinate systems are parallel. The transformation relationship is: Xc = X; Yc = Y; Zc =-H. The negative sign indicates that the optical center of the camera is in the positive direction of the Z-axis in the world coordinate system, and point P is in the negative direction of the Z-axis.

4. The multi-angle projection surgical navigation device based on near-infrared imaging according to claim 3, characterized in that, The coordinate transformation relationship from the camera coordinate system to the camera imaging plane coordinate system specifically includes: According to the principle of perspective projection, the pixel coordinates (Ic_u, Ic_v) of point P in the Ic coordinate system of the camera imaging plane satisfy: [Ic_u; Ic_v; 1] = Kc Substituting [Xc; Yc; Zc] into Xc, Yc, and Zc, we get: Ic_u = (f_c / sx) X / (-H) + u0;Ic_v = (f_c / sy) Y / (-H) + v0; The image data acquired by the infrared camera is the gray value G(Ic_u, Ic_v) corresponding to each pixel (Ic_u, Ic_v).

5. A multi-angle projection surgical navigation device based on near-infrared imaging according to claim 3, characterized in that, The coordinate transformation from the world coordinate system to the projector coordinate system specifically includes: the coordinates of the optical center Pi of the i-th projector in the world coordinate system are determined according to its distribution angle. Let the angle between the i-th projector and the X-axis be α_i = 2π(i-1) / N, the optical axis of the projector point to point O, and the angle between the optical axis and the Z-axis be θ. The coordinates of Pi in the Z-axis direction are L. cosθ, the distance from the projection of the XY plane to point O is L. sinθ; The transformation between the projector coordinate system and the world coordinate system is achieved through the rotation matrix Ri and the translation vector Ti, [Xi;Yi; Zi] = Ri [X - PwX; Y - PwY; 0 - PwZ]; where (PwX, PwY, PwZ) are the world coordinates of Pi, and Ri is a 3×3 rotation matrix.

6. The multi-angle projection surgical navigation device based on near-infrared imaging according to claim 3, characterized in that, The coordinate transformation relationship from the projector coordinate system to the projector imaging plane coordinate system specifically includes: the pixel coordinates (Ii_u, Ii_v) of point P in the Ii coordinate system of the i-th projector imaging plane satisfying: [Ii_u; Ii_v; 1] = Ki [Xi; Yi; Zi].

Images