Conversion relation determination method, system and device, equipment and medium

Abstract

The invention relates to a method, system and device for determining a conversion relation, equipment and a medium. The method comprises the following steps: acquiring an optical image acquired by a target camera for a preset calibration plate and a ray image acquired by X-ray equipment for the preset calibration plate; wherein the visible light angular point and the X-ray angular point of the calibration pattern on the preset calibration plate are the same; performing angular point detection on the optical image to obtain an angular point optical coordinate under a camera coordinate system of the target camera, and performing angular point detection on the ray image to obtain an angular point ray coordinate under a ray coordinate system of the X-ray equipment; and determining a target conversion relation between the camera coordinate system and the ray coordinate system according to the angular point optical coordinates, the angular point ray coordinates and the angular point standard coordinates of the calibration pattern. Therefore, the target camera and the X-ray equipment can track the same angular point, and the conversion relation between the coordinate system where the target camera is located and the coordinate system where the X-ray equipment is located can be efficiently determined.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics, and in particular to a method, system, device, equipment and medium for determining a conversion relationship. Background Art

[0002] The C-arm is an X-ray fluoroscopic imaging device used to assist doctors in determining the spatial position of the device. In the related art, the spatial posture of the C-arm can be measured by the rotation angle measurement function of the C-arm itself. However, the accuracy of the spatial posture determined by the C-arm's own function is low. In order to achieve high-precision measurement of the C-arm's spatial posture, the C-arm needs to be associated with the camera. How to efficiently determine the spatial position relationship between the camera and the C-arm has become a problem that needs to be solved urgently. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a method, system, device, equipment and medium for determining a conversion relationship.

[0004] In a first aspect, the present disclosure provides a method for determining a conversion relationship, the method comprising:

[0005] Acquire an optical image of a preset calibration plate captured by a target camera and a radiographic image of the preset calibration plate captured by an X-ray device; wherein the visible light corner point and the X-ray corner point of the calibration pattern on the preset calibration plate are the same;

[0006] Performing corner point detection on the optical image to obtain optical coordinates of the corner points in the camera coordinate system of the target camera, and performing corner point detection on the ray image to obtain ray coordinates of the corner points in the ray coordinate system of the X-ray device;

[0007] The target conversion relationship between the camera coordinate system and the ray coordinate system is determined according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern.

[0008] In a second aspect, the present disclosure further provides a system for determining a conversion relationship, the system comprising a target camera, a preset calibration plate, an X-ray device, and a computer;

[0009] The target camera is used to acquire an image of the preset calibration plate to obtain an optical image and transmit the optical image to the computer; the X-ray device is used to acquire an image of the preset calibration plate to obtain a radiographic image and transmit the radiographic image to the computer;

[0010] The computer is used to implement the method provided in the first aspect based on the optical image and the radiation image.

[0011] In a third aspect, the present disclosure provides a device for determining a conversion relationship, the device comprising:

[0012] An acquisition module, used to acquire an optical image acquired by a target camera from a preset calibration plate and a radiographic image acquired by an X-ray device from the preset calibration plate; wherein the visible light corner point and the X-ray corner point of the calibration pattern on the preset calibration plate are the same;

[0013] A detection module, configured to perform corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and to perform corner point detection on the radiographic image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device;

[0014] A determination module is used to determine a target conversion relationship between the camera coordinate system and the ray coordinate system according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern.

[0015] In a fourth aspect, the present disclosure further provides an electronic device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When one or more programs are executed by one or more processors, the one or more processors implement the method provided in the first aspect.

[0019] In a fifth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the method provided in the first aspect when executed by a processor.

[0020] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0021] The present invention discloses a method, system, device, equipment and medium for determining a conversion relationship. The method includes: obtaining an optical image captured by a target camera on a preset calibration plate and a radiographic image captured by an X-ray device on the preset calibration plate; wherein the visible light corner points and the X-ray corner points of the calibration pattern on the preset calibration plate are the same; performing corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and performing corner point detection on the radiographic image to obtain the radiographic coordinates of the corner points in the radiographic coordinate system of the X-ray device; and determining the target conversion relationship between the camera coordinate system and the radiographic coordinate system according to the optical coordinates of the corner points, the radiographic coordinates of the corner points and the standard coordinates of the corner points of the calibration pattern. In the above scheme, a preset calibration plate with overlapping corner points under different imaging principles is designed, and corner point detection is performed on the images of the preset calibration plate respectively collected by the target camera and the X-ray device to obtain the optical coordinates and ray coordinates of the corner points, and the conversion relationship between the coordinate system of the target camera and the coordinate system of the X-ray device is determined based on the optical coordinates, ray coordinates and standard coordinates of the corner points. Since the visible light corner points and X-ray corner points of the preset calibration plate are the same, even if the imaging principles of the target camera and the X-ray device are different, the visual features of the collected images are the same, thereby realizing the tracking of the same corner points by the target camera and the X-ray device, and realizing the efficient determination of the conversion relationship between the coordinate system of the target camera and the coordinate system of the X-ray device. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 A flowchart of a method for determining a conversion relationship provided in an embodiment of the present disclosure;

[0025] Figure 2 A schematic diagram of collecting optical images and radiographic images provided in an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of a preset calibration plate provided in an embodiment of the present disclosure;

[0027] Figure 4 A schematic diagram of a connection between a depth camera and a C-arm provided in an embodiment of the present disclosure;

[0028] Figure 5A flowchart of another method for determining a conversion relationship provided in an embodiment of the present disclosure;

[0029] Figure 6 A schematic diagram of a method for determining a conversion relationship provided in an embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of the structure of a device for determining a conversion relationship provided in an embodiment of the present disclosure;

[0031] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0034] The C-arm is an intraoperative imaging device that is widely used in spinal surgery and other surgeries to assist doctors in determining the spatial position of the instrument. The C-arm has a perspective function, and the spatial posture of the C-arm can be measured through the rotation angle measurement function of the C-arm itself. However, the spatial posture determined by the C-arm's own function is of low accuracy.

[0035] In order to achieve high-precision measurement of the C-arm's spatial posture, the C-arm needs to be bound to the camera and the camera combined with the stereoscopic vision tracking algorithm is used to achieve spatial positioning of the C-arm. In order to map the camera's tracking posture to the coordinate system where the C-arm is located, it is necessary to obtain the spatial position relationship between the camera and the C-arm. How to obtain the spatial position relationship between the camera and the C-arm has become a problem that needs to be solved urgently.

[0036] In order to solve at least one of the above technical problems, the method for determining the conversion relationship provided by the embodiment of the present disclosure is described below. In the embodiment of the present disclosure, the method for determining the conversion relationship can be performed by an electronic device. The electronic device can include a device with communication functions such as a tablet computer, a desktop computer, a laptop computer, etc., and can also include a device simulated by a virtual machine or a simulator.

[0037] Figure 1 FIG. 1 is a flow chart showing a method for determining a conversion relationship provided by an embodiment of the present disclosure. Figure 1 As shown, the method for determining the conversion relationship may include the following steps.

[0038] Step 101, obtaining an optical image of a preset calibration plate captured by a target camera and a radiographic image of a preset calibration plate captured by an X-ray device; wherein the visible light corner points and X-ray corner points of the calibration pattern on the preset calibration plate are the same.

[0039] The target camera may be a camera whose spatial conversion relationship with the X-ray device is to be determined, the target camera may be a camera used to assist the X-ray device in completing a specific function, and the target camera may be a camera with a three-dimensional coordinate measurement function. For example, in the process of intraoperative navigation of an instrument through an X-ray device, the target camera may be used to determine the angle difference of the X-ray device when it rotates to different imaging positions, and / or to capture images of the instrument. The optical image may be an image formed by recording light in the visible light band, and the optical image may be an image directly captured by the target camera, or an image after the directly captured image is grayed.

[0040] The X-ray device may be an instrument that uses X-rays for imaging. In some embodiments of the present disclosure, the X-ray device may be a multi-position imaging device, and the multi-position imaging device may be a device that performs medical imaging of a target object at different positions. Specifically, the multi-position imaging device may rotate around the target object and stop at different angles to perform X-ray imaging of the target object. This embodiment does not limit the type of the multi-position imaging device. For example, the multi-position imaging device may be a C-arm or an O-arm, etc. The radiation image may be an image formed after the X-ray penetrates the object.

[0041] The preset calibration plate can be used to calibrate the spatial conversion relationship between a device based on visible light imaging and a device based on X-ray imaging. The preset calibration plate is also called a calibration plate. The preset calibration plate can be formed by stacking or splicing material layers of different colors and different developing materials. The calibration pattern can be a pattern on the preset calibration plate, the calibration pattern can be a pattern with known pattern features, and the calibration pattern can be formed by stacking or splicing material layers.

[0042] Corner points may be points in a pattern where the grayscale changes suddenly along different directions (e.g., horizontal and vertical directions). Visible light corner points may be corner points set in a calibration pattern based on visible light imaging characteristics. X-ray corner points may be corner points set in a calibration pattern based on X-ray imaging characteristics. It is understandable that both the visible light corner points and the X-ray corner points are pre-designed corner points on a preset calibration plate. The visible light corner point and the X-ray corner point being the same may include the visible light corner point and the X-ray corner point coinciding, or the distance between the visible light corner point and the X-ray corner point being less than a preset distance threshold. It is understandable that due to limitations such as manufacturing processes, it is difficult to achieve complete coincidence of the visible light corner point and the X-ray corner point, and therefore, when the distance between the visible light corner point and the X-ray corner point is less than a preset distance threshold, the visible light corner point and the X-ray corner point may be approximately considered to be the same.

[0043] In the disclosed embodiment, the target camera and the X-ray device can be relatively stationary, and the preset calibration plate is placed in the common field of view of the target camera and the X-ray device. The X-ray device collects images of the preset calibration plate to generate a radiographic image. When the X-ray device does not move, the target camera collects images of the preset calibration plate to generate an optical image.

[0044] Figure 2 A schematic diagram of collecting optical images and radiographic images provided in an embodiment of the present disclosure, such as Figure 2 As shown, the preset calibration plate is placed in the common field of view of the target camera and the X-ray device, and the target camera and the X-ray device simultaneously image the preset calibration plate to obtain an optical image and a radiographic image, respectively.

[0045] In some embodiments of the present disclosure, the preset calibration plate includes a radiation developing layer of a first color and a radiation non-developing layer of a second color, the radiation developing layer constructs a calibration pattern in a hollow manner on the upper layer of the radiation non-developing layer, and the thickness of the radiation developing layer is not greater than a thickness threshold.

[0046] Among them, the first color and the second color can be different colors, and the color difference between the first color and the second color can be greater than a preset color difference threshold. For example, the first color and the second color can be black and white, respectively. The ray developing layer can be a planar layer made of a developing material, and the developing material can be a material that develops under X-rays. For example, the developing material can be a metal such as a stainless steel material. The ray non-developing layer can be a planar layer made of a non-developing material, and the non-developing material can be a non-metallic material that does not develop under X-rays. The thickness threshold can be a preset maximum thickness of the ray developing layer, and this embodiment does not limit the thickness threshold. By limiting the thickness of the ray developing layer, the imaging edge of the ray image near the X-ray corner point can be made clearer, so that the ray coordinates of the corner point identified subsequently can be more accurate. The upper layer can be the layer closer to the ray emitting component of the two layers.

[0047] In this embodiment, the radiation developing layer is superimposed on the radiation non-developing layer in a hollow manner to form a calibration pattern, or the radiation developing layer and the radiation non-developing layer can be spliced ​​on the same plane layer to form a calibration pattern. Since the radiation developing layer and the radiation non-developing layer have different colors, the points where the visible light color suddenly changes along different directions are visible light corner points. Since the radiation developing layer and the radiation non-developing layer have different developing properties to X-rays, the colors of the radiation developing layer and the radiation non-developing layer are different under X-rays, and the points where the X-ray developing color suddenly changes along different directions are X-ray corner points. It can be understood that since the colors and developing properties of the two layers are different, the positions of the visible light corner points and the X-ray corner points in the calibration pattern are the same.

[0048] In an optional embodiment, the preset calibration plate includes a top layer and a bottom layer, the radiation non-developing layer of the bottom layer is a non-metallic material, the thickness can be 10 mm, the radiation non-developing layer does not develop under X-rays, and the color can be pure white. The radiation developing layer of the top layer can be stainless steel 316L, the thickness can be 0.1 mm, and the color can be pure black. By using high-density metal, the radiation developing layer can be developed under X-rays. The radiation developing layer is hollowed out according to a checkerboard pattern. The radiation developing layer is bonded to the radiation non-developing layer to obtain a preset calibration plate, the surface of the preset calibration plate is flat, and it cannot fall off after high-temperature sterilization. Optionally, the color of the above-mentioned radiation non-developing layer can be pure black, and the color of the radiation non-developing layer can be pure white.

[0049] Figure 3 A schematic diagram of a preset calibration plate provided in an embodiment of the present disclosure, such as Figure 3 As shown, the calibration pattern of the preset calibration plate is a checkerboard pattern. Since the visible light corner points and X-ray corner points of the calibration pattern on the preset calibration plate are the same, if the target camera and the X-ray device are located at the same position, the image features of the image captured for the calibration pattern are theoretically the same.

[0050] In the above solution, the two plane layers constituting the preset calibration plate are set to different colors and developability, so that the visible light corner points and X-ray corner points of the preset calibration plate are set to coincide with each other.

[0051] In some embodiments of the present disclosure, the target camera is a depth camera, the X-ray device is a C-arm, the target camera is rigidly connected to the X-ray device, and the angle between the target camera and the ray emitting component in the X-ray device is a preset angle.

[0052] Among them, the depth camera can be used to measure the distance between itself and the photographed object, that is, the depth camera can be used to obtain the depth information of the scene, and the depth camera can be a structured light camera or a binocular depth camera. The C-arm is also called a C-arm machine. The ray emitting component can be a functional component for generating X-rays in a multi-position imaging device. The ray emitting component is used in conjunction with the image receiving component, and the image receiving component can be a component in the multi-position imaging device for receiving X-rays that pass through the target object.

[0053] The preset angle may be the angle between a first line and a second line that are preset, the first line may be the connection between the ray emitting assembly and the rotation center point of the C-arm, and the second line may be the connection between the target camera and the rotation center point. The preset angle may be half of the predicted rotation angle, and the predicted rotation angle may be the estimated angle difference between the two working positions to which the C-arm is rotated. For example, if the working positions include the anteroposterior position and the lateral position, and the predicted rotation angles of the anteroposterior position and the lateral position are 90°, the preset angle may be 45°.

[0054] Figure 4 A schematic diagram of a connection between a depth camera and a C-arm provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, in this embodiment, the ray emitting component and the image receiving component are respectively located at the two ends of the arc arm, which is also called a cantilever. The depth camera is rigidly connected to the arc arm of the C-arm, and the angle between the depth camera and the ray emitting component of the C-arm is a preset angle.

[0055] In the above scheme, when the predicted rotation angle between the two imaging positions of the C-arm is large, by setting the angle between the target camera and the ray emitting assembly to a preset angle, the observation angle of the target camera will not be too large in different subsequent imaging positions, thereby reducing the distortion of the observed image.

[0056] In some embodiments of the present disclosure, preset calibration plates are placed at multiple measurement positions within the common field of view of the target camera and the X-ray device, respectively. The measurement positions correspond one-to-one to image pairs, and an image pair includes an optical image and a radiographic image.

[0057] The common field of view may be the area captured by the target camera and the X-ray device at the same time. The measurement position may be the position where the target camera and the X-ray device respectively image the preset calibration plate, and the different measurement positions may be positions at different distances from the target camera and the X-ray device. This embodiment does not limit the number of measurement positions, for example, the number of measurement positions may be 20.

[0058] In this embodiment, the preset calibration plate is placed in advance at different measurement positions in the common field of view, and at each measurement position, the target camera collects an optical image of the preset calibration plate, and the X-ray device collects a radiographic image of the preset calibration plate, and an image pair is formed based on the optical image and the radiographic image. In the above scheme, by setting multiple measurement positions and collecting multiple optical images and radiographic images of the preset calibration plate, multiple target conversions can be determined, and the final conversion relationship is subsequently determined based on the multiple target conversion relationships, thereby improving the accuracy of the conversion relationship.

[0059] Step 102, performing corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and performing corner point detection on the radiographic image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device.

[0060] Among them, the camera coordinate system may be a coordinate system established based on the target camera. The optical coordinates of the corner point may be the two-dimensional coordinates of the visible light corner point recorded in the optical image. The present embodiment does not limit the accuracy of the optical coordinates of the corner point. For example, the optical coordinates of the corner point may be sub-pixel accuracy. The ray coordinate system may be a coordinate system established based on an X-ray device. The ray coordinates of the corner point may be the two-dimensional coordinates of the X-ray corner point recorded in the ray image. The present embodiment does not limit the accuracy of the ray coordinates of the corner point. For example, the ray coordinates of the corner point may be sub-pixel accuracy.

[0061] In the disclosed embodiment, the conversion relationship determination device can perform corner point recognition on the optical image by using a preset corner point recognition algorithm to obtain the optical coordinates of the corner points in the camera coordinate system. In addition, the conversion relationship determination device can perform corner point recognition on the ray image by using a preset corner point recognition algorithm to obtain multiple ray coordinates of the corner points. This embodiment does not limit the corner point recognition algorithm. For example, the corner point recognition algorithm can be an algorithm based on a neural network, an algorithm based on image grayscale, etc.

[0062] Optionally, after determining the corner point optical coordinates and the corner point ray coordinates, the conversion relationship determination device may mark the corner point optical coordinates on the optical image to obtain a first marked image, and display the first marked image. The user browses the first marked image and performs a deletion operation on the optical coordinates to be deleted in the corner point optical coordinates. The optical coordinates to be deleted may be the optical coordinates of the corner point that have problems such as abnormal order or do not correspond to the actual corner point. In response to the user's deletion operation on the optical coordinates to be deleted on the first marked image, the optical coordinates to be deleted are deleted.

[0063] Furthermore, the device for determining the conversion relationship can annotate the ray optical coordinates on the ray image to obtain a second annotated image, and display the second annotated image. The user browses the second annotated image and performs a deletion operation on the ray coordinates to be deleted in the corner point ray coordinates. The ray coordinates to be deleted may be the corner point ray coordinates with problems such as abnormal sequence or non-correspondence with the actual corner point. In response to the user's deletion operation on the ray coordinates to be deleted on the second annotated image, the ray coordinates to be deleted are deleted. Thus, by eliminating abnormal coordinates, the influence of noise and imaging quality on the final determined conversion relationship is reduced.

[0064] Step 103, determining a target conversion relationship between the camera coordinate system and the ray coordinate system according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern.

[0065] Among them, the standard coordinates of the corner points can be the gold standard of the corner points in the calibration pattern, and the standard coordinates of the corner points can be the three-dimensional coordinates in the calibration coordinate system. The calibration coordinate system can be the coordinate system where the preset calibration plate is located. The target transformation relationship can characterize the spatial position transformation between the camera coordinate system and the ray coordinate system, and the target transformation relationship can include a rotation transformation relationship and a translation transformation relationship. The rotation transformation relationship can be represented by a 3x3 rotation matrix, and the translation transformation relationship can be represented by a 3x1 translation vector.

[0066] In the disclosed embodiment, the conversion relationship determination device can obtain the internal parameters of the target camera, the internal parameters of the X-ray device, the optical coordinates of the corner point, the ray coordinates of the corner point, and the standard coordinates of the corner point, and calculate the spatial transformation between the camera coordinate system and the ray coordinate system according to the above parameters to obtain the target conversion relationship. Optionally, the target conversion relationship can be recorded in a preset file, and the preset file can be in an extensible markup language (XML) format.

[0067] In the process of determining the target transformation relationship, the transformation relationship first determined according to the internal parameters of the target camera, the internal parameters of the X-ray device, the optical coordinates of the corner point, the ray coordinates of the corner point, and the standard coordinates of the corner point can be used as the intermediate transformation relationship, and the intermediate transformation relationship is adjusted by optimizing the reprojection error of the camera image and the ray image until the reprojection error of the intermediate transformation relationship is less than the preset error threshold, and the intermediate transformation relationship is determined as the target transformation relationship. And the last determined reprojection error can also be recorded in the preset file.

[0068] Figure 5 A flow chart of another method for determining a conversion relationship provided in an embodiment of the present disclosure is shown as follows: Figure 5As shown, in some embodiments of the present disclosure, according to the optical coordinates of the corner points, the ray coordinates of the corner points, and the standard coordinates of the corner points of the calibration pattern, the target conversion relationship between the camera coordinate system and the ray coordinate system is determined, including:

[0069] Step 501 : determining a first conversion relationship between a camera coordinate system and a calibration coordinate system where a calibration pattern is located according to the optical coordinates of the corner points, the internal parameters of the target camera, and the standard coordinates of the corner points.

[0070] The internal parameters of the target camera may be parameters describing the imaging characteristics of the target camera itself, and the internal parameters may include one or more of the focal length, the projection center, and the deformation coefficient. The internal parameters of the target camera may be pre-calibrated, and the internal parameters of the target camera may be represented by a 3x3 matrix. The first transformation relationship may characterize the spatial position transformation between the camera coordinate system and the calibration coordinate system.

[0071] In this embodiment, the conversion relationship determination device can perform spatial transformation calculation on the optical coordinates of the corner point, the intrinsic parameters of the target camera, and the standard coordinates of the corner point according to a preset coordinate system transformation algorithm to obtain a first transformation relationship between the camera coordinate system and the calibration coordinate system. The coordinate system transformation algorithm can be used to determine the spatial transformation between corresponding coordinate systems based on the two-dimensional coordinates and three-dimensional coordinates of the same corner point, and this embodiment does not limit the coordinate system transformation algorithm. For example, the coordinate system transformation algorithm can be a perspective point (Perspective-n-Point, PnP) algorithm.

[0072] Step 502, determining a second conversion relationship between the calibration coordinate system and the ray coordinate system according to the ray coordinates of the corner point, the internal parameters of the X-ray device, and the standard coordinates of the corner point.

[0073] The internal parameters of the X-ray device may be parameters describing the imaging characteristics of the X-ray device itself, the internal parameters of the X-ray device may be pre-calibrated, and the internal parameters of the X-ray device may be represented by a 3x3 matrix. The second transformation relationship may characterize the spatial position transformation between the ray coordinate system and the calibration coordinate system.

[0074] In this embodiment, the device for determining the conversion relationship can perform spatial transformation calculations on the corner point ray coordinates, the internal parameters of the X-ray device, and the corner point standard coordinates according to a preset coordinate system conversion algorithm to obtain a second conversion relationship between the ray coordinate system and the calibration coordinate system.

[0075] Step 503: Perform relationship conversion according to the first conversion relationship and the second conversion relationship to determine a target conversion relationship.

[0076] In this embodiment, after determining the first conversion relationship and the second conversion relationship, the conversion relationship determination device may perform a composite transformation operation on the first conversion relationship and the second conversion relationship to obtain a target conversion relationship.

[0077] In the above scheme, the calibration coordinate system is used as a bridge to efficiently determine the conversion relationship between the camera coordinate system and the ray coordinate system.

[0078] The method for determining the conversion relationship of the embodiment of the present disclosure obtains an optical image collected by a target camera on a preset calibration plate and a radiographic image collected by an X-ray device on the preset calibration plate; wherein the visible light corner points and the X-ray corner points of the calibration pattern on the preset calibration plate are the same; corner point detection is performed on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and corner point detection is performed on the radiographic image to obtain the radiographic coordinates of the corner points in the radiographic coordinate system of the X-ray device; the target conversion relationship between the camera coordinate system and the radiographic coordinate system is determined according to the optical coordinates of the corner points, the radiographic coordinates of the corner points and the standard coordinates of the corner points of the calibration pattern. In the above scheme, a preset calibration plate with overlapping corner points under different imaging principles is designed, and corner point detection is performed on the images of the preset calibration plate respectively collected by the target camera and the X-ray device to obtain the optical coordinates and ray coordinates of the corner points, and the conversion relationship between the coordinate system of the target camera and the coordinate system of the X-ray device is determined based on the optical coordinates, ray coordinates and standard coordinates of the corner points. Since the visible light corner points and X-ray corner points of the preset calibration plate are the same, even if the imaging principles of the target camera and the X-ray device are different, the visual features of the collected images are the same, thereby realizing the tracking of the same corner points by the target camera and the X-ray device, and realizing the efficient determination of the conversion relationship between the coordinate system of the target camera and the coordinate system of the X-ray device.

[0079] In some embodiments of the present disclosure, before performing corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, the method for determining the conversion relationship also includes: performing grayscale processing on the optical image to obtain a grayscale image, and updating the optical image based on the grayscale image.

[0080] In this embodiment, the conversion relationship determination device can grayscale the optical image according to the grayscale algorithm to generate a grayscale image, and use the grayscale image to replace the original optical image. In the above scheme, by grayscale processing the optical image, the subsequent corner point detection based on the optical image is more accurate.

[0081] In some embodiments of the present disclosure, before performing corner point detection on a radiation image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device, the method for determining the conversion relationship also includes: adjusting the window width and window position of the radiation image until the contrast of the radiation image reaches a contrast threshold.

[0082] Among them, the window width can be used to determine the size of the grayscale range displayed in the image. When the window width is widened, the grayscale range displayed in the image is larger and the contrast is lower; when the window width is narrowed, the grayscale range displayed in the image is smaller and the contrast is higher. The window level can be used to determine the center position of the grayscale in the image. When the window level is increased, the image becomes brighter as a whole; when the window level is decreased, the image becomes darker as a whole. Contrast can represent the degree of brightness difference in the image. The contrast threshold can be the minimum value of the contrast.

[0083] In this embodiment, after determining the radiographic image, the conversion relationship determination device can adjust the window width and window position of the radiographic image at least once, and determine the contrast of the radiographic image after each adjustment, and determine whether the contrast reaches the contrast threshold, if so, stop adjusting, otherwise, continue adjusting the window width and / or window position. In the above scheme, by adjusting the window width and window position of the radiographic image, the image boundary of the radiographic image is clearer, and the accuracy of corner point detection based on the radiographic image is improved.

[0084] Optionally, after generating the grayscale image, the conversion relationship determination device may adjust the window width and window position of the grayscale image until the contrast of the grayscale image reaches a contrast threshold, thereby making the boundaries of the corner points recorded in the grayscale image more obvious.

[0085] In some embodiments of the present disclosure, the method for determining the transformation relationship also includes: mapping the standard coordinates of the corner point to the camera coordinate system according to the first transformation relationship to obtain the first transformation coordinates, and mapping the standard coordinates of the corner point to the ray coordinate system according to the second transformation relationship to obtain the second transformation coordinates; determining whether the target transformation relationship passes the error detection based on the first transformation coordinates, the corner point optical coordinates, the second transformation coordinates, and the corner point ray coordinates.

[0086] In this embodiment, the device for determining the transformation relationship can transform the standard coordinates of the corner point from the calibration coordinate system to the camera coordinate system according to the first transformation relationship to obtain the first transformation coordinates. The distance between the first transformation coordinates and the optical coordinates of the corner point corresponding to the same corner point is calculated to obtain multiple first error distances. In addition, the device for determining the transformation relationship can transform the standard coordinates of the corner point from the calibration coordinate system to the ray coordinate system according to the second transformation relationship to obtain the second transformation coordinates. The distance between the second transformation coordinates and the ray coordinates of the corner point corresponding to the same corner point is calculated to obtain multiple second error distances. If the multiple first error distances and the multiple second error distances are not greater than the preset error threshold, it means that the accuracy of the first transformation relationship and the second transformation relationship is high, and thus the target transformation relationship determined based on the first transformation relationship and the second transformation relationship passes the error detection.

[0087] If there is at least one error distance greater than the preset error threshold, the corresponding conversion relationship is adjusted according to the error distance greater than the preset error threshold. The error distance includes the first error distance and / or the second error distance. Until the error distance calculated according to the adjusted conversion relationship is not greater than the preset error threshold, and a new target conversion relationship is determined according to the adjusted conversion relationship.

[0088] Specifically, if there is at least one first error distance greater than a preset error threshold, the first conversion relationship is adjusted according to the first error distance greater than the preset error threshold, until the error distance calculated according to the adjusted first conversion relationship is not greater than the preset error threshold, and a new target conversion relationship is determined according to the adjusted first conversion relationship. If there is at least one second error distance greater than the preset error threshold, the second conversion relationship is adjusted according to the second error distance greater than the preset error threshold, until the error distance calculated according to the adjusted second conversion relationship is not greater than the preset error threshold, and a new target conversion relationship is determined according to the adjusted second conversion relationship. If there is at least one first error distance greater than the preset error threshold, and at least one second error distance greater than the preset error threshold, the first conversion relationship is adjusted according to the first error distance greater than the preset error threshold, until the error distance calculated according to the adjusted first conversion relationship is not greater than the preset error threshold, and the second conversion relationship is adjusted according to the second error distance greater than the preset error threshold, until the second error distance calculated according to the adjusted conversion relationship is not greater than the preset error threshold. Further, a new target conversion relationship is determined according to the adjusted first conversion relationship and the second conversion relationship.

[0089] In the above scheme, whether the first conversion relationship is accurate is determined based on the first conversion relationship, the standard coordinates of the corner point, and the optical coordinates of the corner point, and whether the second conversion relationship is accurate is determined based on the second conversion relationship, the standard coordinates of the corner point, and the ray coordinates of the corner point, thereby realizing closed-loop adjustment and improving the accuracy of the target conversion relationship finally determined.

[0090] The disclosed embodiment also provides a system for determining a conversion relationship, the system comprising a target camera, a preset calibration plate, an X-ray device, and a computer; the target camera is rigidly connected to the X-ray device.

[0091] The target camera is used to acquire an image of a preset calibration plate to obtain an optical image and transmit the optical image to a computer. The X-ray device is used to acquire an image of a preset calibration plate to obtain a radiographic image and transmit the radiographic image to a computer. The computer is used to implement a method for determining the above-mentioned conversion relationship based on the optical image and the radiographic image.

[0092] The computer may be a computer for implementing multi-view navigation of intraoperative instruments, and the method for determining the conversion relationship may be a sub-function of implementing multi-view navigation.

[0093] Next, the method for determining the conversion relationship in the embodiment of the present disclosure is further explained through a specific example. Figure 6 A schematic diagram of a method for determining a conversion relationship provided in an embodiment of the present disclosure, in which the X-ray device is a C-arm, such as Figure 6 As shown, the method for determining the conversion relationship includes:

[0094] First, calibrate the target camera and the C-arm. Specifically, perform internal parameter calibration on the target camera and the C-arm, and the internal parameters include focal length, projection center, deformation coefficient, etc. Further, perform image acquisition. Specifically, place a preset calibration plate in the common field of view of the target camera and the C-arm, and image the preset calibration plate simultaneously through the target camera and the C-arm to obtain the optical image collected by the target camera and the radiographic image collected by the C-arm. Place the preset calibration plate at multiple different measurement positions, and repeatedly acquire about 20 image pairs, each image pair including an optical image and a radiographic image.

[0095] Furthermore, the window width and window position are adjusted. Specifically, for each image pair, the device for determining the conversion relationship can adjust the window width and window position of the ray image in the image pair until a high-contrast ray image is obtained. For the optical image, grayscale processing is performed. Optionally, the window width and window position of the optical image after the grayscale processing can be adjusted until a high-contrast optical image is obtained. Furthermore, corner point detection is performed. Specifically, corner point detection is performed on the optical image and the ray image respectively, and the optical coordinates and ray coordinates of the corner points with sub-pixel detection accuracy are obtained. The optical coordinates and ray coordinates of the corner points are both two-dimensional coordinates. Furthermore, the corner points are removed. Specifically, due to the influence of noise and imaging quality, not every corner point in the image can be detected correctly. Optionally, the user can visually browse the coordinates corresponding to the corner points and remove abnormal coordinates such as outliers.

[0096] Furthermore, stereo calibration is performed. The device for determining the transformation relationship can calculate the spatial transformation from the camera coordinate system to the ray coordinate system by optimizing the reprojection error according to the optical coordinates of the corner points, the ray coordinates of the corner points, the standard coordinates of the corner points of the preset calibration plate, the internal parameters of the target camera, and the internal parameters of the C-arm, and obtain the target transformation relationship. Specifically, the first transformation relationship from the calibration coordinate system to the camera coordinate system is calculated according to the optical coordinates of the corner points, the internal parameters of the target camera, and the standard coordinates of the corner points; and the second transformation relationship from the calibration coordinate system to the ray coordinate system is calculated according to the ray coordinates of the corner points, the internal parameters of the X-ray device, and the standard coordinates of the corner points. The initial intermediate transformation relationship is determined according to the first transformation relationship and the second transformation relationship, and the intermediate transformation relationship is optimized by optimizing the reprojection error to the optical image and the ray image to obtain the optimized target transformation relationship. The target transformation relationship, the final reprojection error, the internal parameters of the target camera, and the internal parameters of the X-ray device can be recorded in a preset file.

[0097] In the above scheme, based on the different imaging principles of the target camera and the X-ray device, a preset calibration plate is designed that can perform both perspective imaging through the X-ray device and visible light imaging through the target camera, and the visible light corner points and X-ray corner points designed on the preset calibration plate are the same, presenting consistent visual features. Then, based on the preset calibration plate, the calibration of the coordinate system of the X-ray device to the coordinate system of the target camera is realized.

[0098] Figure 7 A schematic diagram of the structure of a device for determining a conversion relationship provided by an embodiment of the present disclosure is shown.

[0099] like Figure 7 As shown, the conversion relationship determination device 700 may include:

[0100] An acquisition module 701 is used to acquire an optical image acquired by a target camera from a preset calibration plate and a radiographic image acquired by an X-ray device from the preset calibration plate; wherein the visible light corner point and the X-ray corner point of the calibration pattern on the preset calibration plate are the same;

[0101] A detection module 702 is used to perform corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and to perform corner point detection on the radiographic image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device;

[0102] The determination module 703 is used to determine the target conversion relationship between the camera coordinate system and the ray coordinate system according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern.

[0103] Optionally, the preset calibration plate includes a radiation developing layer of a first color and a radiation non-developing layer of a second color, the radiation developing layer constructs the calibration pattern in a hollowed-out manner on an upper layer of the radiation non-developing layer, and a thickness of the radiation developing layer is not greater than a thickness threshold.

[0104] Optionally, the target camera is a depth camera, the X-ray device is a C-arm, the target camera is rigidly connected to the X-ray device, and the angle between the target camera and an emitting component in the X-ray device is a preset angle.

[0105] Optionally, the conversion relationship determination device 700 further includes:

[0106] A grayscale module, used for performing grayscale processing on the optical image to obtain a grayscale image before performing corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and updating the optical image according to the grayscale image;

[0107] The adjustment module is used to adjust the window width and window position of the radiation image before performing corner point detection on the radiation image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device, until the contrast of the radiation image reaches a contrast threshold.

[0108] Optionally, the determining module 703 is used to:

[0109] Determine a first conversion relationship between the camera coordinate system and the calibration coordinate system where the calibration pattern is located according to the optical coordinates of the corner point, the internal parameters of the target camera, and the standard coordinates of the corner point;

[0110] Determining a second conversion relationship between the calibration coordinate system and the ray coordinate system according to the corner point ray coordinates, the internal parameters of the X-ray device, and the corner point standard coordinates;

[0111] Relationship conversion is performed according to the first conversion relationship and the second conversion relationship to determine the target conversion relationship.

[0112] Optionally, the conversion relationship determination device 700 further includes:

[0113] A mapping module, used for mapping the standard coordinates of the corner point to the camera coordinate system according to the first transformation relationship to obtain first transformation coordinates, and mapping the standard coordinates of the corner point to the ray coordinate system according to the second transformation relationship to obtain second transformation coordinates;

[0114] A detection module is used to determine whether the target transformation relationship passes the error detection according to the first transformation coordinates, the corner point optical coordinates, the second transformation coordinates, and the corner point ray coordinates.

[0115] It should be noted that Figure 7 The illustrated conversion relationship determination device 700 can execute each step in the above conversion relationship determination method embodiment, and realize each process and effect in the above conversion relationship determination method embodiment, which will not be elaborated here.

[0116] Figure 8 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0117] like Figure 8 As shown, the electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0118] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0119] The memory 802 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 802 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 802 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 802 is a non-volatile solid-state memory. In a particular embodiment, the memory 802 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0120] The processor 801 reads and executes the computer program instructions stored in the memory 802 to perform the steps of the method for determining the conversion relationship provided in the embodiment of the present disclosure.

[0121] In one example, the electronic device may further include a transceiver 803 and a bus 804. Figure 8 As shown, the processor 801, the memory 802 and the transceiver 803 are connected via a bus 804 and communicate with each other.

[0122] The bus 804 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 804 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0123] The following is an embodiment of a computer-readable storage medium provided in an embodiment of the present disclosure. The computer-readable storage medium and the method for determining the conversion relationship in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, reference can be made to the embodiment of the method for determining the conversion relationship in the above-mentioned embodiments.

[0124] This embodiment provides a storage medium including computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to execute a method for determining a conversion relationship.

[0125] Of course, the storage medium containing computer executable instructions provided by the embodiment of the present disclosure is not limited to the above method operations, and the computer executable instructions can also execute related operations in the method for determining the conversion relationship provided by any embodiment of the present disclosure.

[0126] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present disclosure can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a computer cloud platform (which can be a personal computer, a server, or a network cloud platform, etc.) to execute the method for determining the conversion relationship provided in each embodiment of the present disclosure.

[0127] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A method for determining a conversion relationship, characterized in that: include: Acquire an optical image of a preset calibration plate captured by a target camera and a radiographic image of the preset calibration plate captured by an X-ray device; wherein the visible light corner point and the X-ray corner point of the calibration pattern on the preset calibration plate are the same; Performing corner point detection on the optical image to obtain optical coordinates of the corner points in the camera coordinate system of the target camera, and performing corner point detection on the ray image to obtain ray coordinates of the corner points in the ray coordinate system of the X-ray device; The target conversion relationship between the camera coordinate system and the ray coordinate system is determined according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern.

2. The method according to claim 1, characterized in that The preset calibration plate includes a radiation developing layer of a first color and a radiation non-developing layer of a second color, wherein the radiation developing layer is hollowed out on the upper layer of the radiation non-developing layer to construct the calibration pattern, and the thickness of the radiation developing layer is not greater than a thickness threshold.

3. The method according to claim 1, characterized in that The target camera is a depth camera, the X-ray device is a C-arm, the target camera is rigidly connected to the X-ray device, and the angle between the target camera and the transmitting component in the X-ray device is a preset angle.

4. The method according to claim 1, characterized in that: Before performing corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, the method further includes: grayscale the optical image to obtain a grayscale image, and update the optical image according to the grayscale image; Before performing corner point detection on the radiographic image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device, the method further includes: The window width and window level of the radiographic image are adjusted until the contrast of the radiographic image reaches a contrast threshold.

5. The method according to claim 1, characterized in that The determining of the target conversion relationship between the camera coordinate system and the ray coordinate system according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern includes: Determine a first conversion relationship between the camera coordinate system and the calibration coordinate system where the calibration pattern is located according to the optical coordinates of the corner point, the internal parameters of the target camera, and the standard coordinates of the corner point; Determining a second conversion relationship between the calibration coordinate system and the ray coordinate system according to the corner point ray coordinates, the internal parameters of the X-ray device, and the corner point standard coordinates; Relationship conversion is performed according to the first conversion relationship and the second conversion relationship to determine the target conversion relationship.

6. The method according to claim 5, characterized in that The method further comprises: Mapping the standard coordinates of the corner point to the camera coordinate system according to the first transformation relationship to obtain first transformation coordinates, and mapping the standard coordinates of the corner point to the ray coordinate system according to the second transformation relationship to obtain second transformation coordinates; Determine whether the target transformation relationship passes error detection according to the first transformation coordinates, the corner point optical coordinates, the second transformation coordinates, and the corner point ray coordinates.

7. A system for determining a conversion relationship, characterized in that: Includes target camera, preset calibration plate, X-ray equipment, computer; The target camera is used to acquire an image of the preset calibration plate to obtain an optical image and transmit the optical image to the computer; the X-ray device is used to acquire an image of the preset calibration plate to obtain a radiographic image and transmit the radiographic image to the computer; The computer is used to implement the method according to any one of claims 1 to 6 based on the optical image and the radiographic image.

8. A device for determining a conversion relationship, characterized in that: include: An acquisition module, used to acquire an optical image acquired by a target camera from a preset calibration plate and a radiographic image acquired by an X-ray device from the preset calibration plate; wherein the visible light corner point and the X-ray corner point of the calibration pattern on the preset calibration plate are the same; A detection module, configured to perform corner point detection on the optical image to obtain the optical coordinates of the corner points in the camera coordinate system of the target camera, and to perform corner point detection on the radiographic image to obtain the ray coordinates of the corner points in the ray coordinate system of the X-ray device; A determination module is used to determine a target conversion relationship between the camera coordinate system and the ray coordinate system according to the corner point optical coordinates, the corner point ray coordinates, and the corner point standard coordinates of the calibration pattern.

9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 6.

Images