Instrument labeling method, system, device, equipment and medium

Abstract

The invention relates to an instrument labeling method, system and device, equipment and a medium. The method comprises the steps that an instrument image collected by a positioning camera for a target instrument is acquired, and a first perspective image obtained by perspective of a target object at a first imaging position by target imaging equipment is acquired; wherein the target instrument is provided with an instrument calibration pattern; performing angular point identification on the instrument image to obtain an initial angular point coordinate under a camera coordinate system of the positioning camera; determining first mark information of the target instrument on the first perspective image according to the initial angular point coordinates and a coordinate system conversion relation; marking the target instrument on the first perspective image according to the first marking information to obtain a first marking image. Therefore, the marking reliability is improved in an automatic marking mode, and a user can visually observe the position of the target instrument on the marked image, so that the user can obtain corresponding image feedback when operating the target instrument.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics, and in particular to an instrument labeling method, system, device, equipment and medium. Background Art

[0002] C-arm is an imaging device used to assist doctors in determining the spatial position of an instrument. In related technologies, 2D C-arm lacks 3D depth information, requiring users to build a corresponding 3D model in their minds based on the 2D perspective image, and determine the position of the instrument after it acts on the object based on the 3D model. However, this method of manually determining the position of the instrument relies on user experience and has low reliability. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides an instrument labeling method, system, device, equipment and medium.

[0004] In a first aspect, the present disclosure provides a device labeling method, the method comprising:

[0005] Acquire an instrument image captured by a positioning camera of a target instrument, and acquire a first perspective image obtained by a target imaging device perspectively observing a target object at a first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is the object acted upon by the target instrument;

[0006] Performing corner point recognition on the device image to obtain initial corner point coordinates in the camera coordinate system of the positioning camera;

[0007] Determine the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship; wherein the coordinate system conversion relationship is the conversion relationship between the camera coordinate system and the device coordinate system of the target imaging device;

[0008] The target device is marked on the first perspective image according to the first marking information to obtain a first marked image.

[0009] In a second aspect, the present disclosure further provides an instrument annotation system, the system comprising a computer, a positioning camera, a target instrument, and a target imaging device;

[0010] The positioning camera is used to acquire an image of the target device to obtain a device image, and transmit the device image to the computer; the target imaging device is used to perform perspective imaging on the target object at a first imaging position to obtain a first perspective image, and transmit the first perspective image to the computer;

[0011] The computer is used to implement the method provided in the first aspect based on the instrument image and the first perspective image.

[0012] In a third aspect, the present disclosure provides an instrument labeling device, the device comprising:

[0013] A first acquisition module is used to acquire an instrument image captured by a positioning camera on a target instrument, and to acquire a first perspective image obtained by a target imaging device perspectively viewing a target object at a first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is an object acted upon by the target instrument;

[0014] A recognition module, used for performing corner point recognition on the device image to obtain initial corner point coordinates in the camera coordinate system of the positioning cameras;

[0015] A conversion module, used to determine the first marking information of the target device on the first perspective image according to the initial corner point coordinates and a coordinate system conversion relationship; wherein the coordinate system conversion relationship is a conversion relationship between the camera coordinate system and the device coordinate system of the target imaging device;

[0016] The first marking module is used to mark the target device on the first perspective image according to the first marking information to obtain a first marked image.

[0017] In a fourth aspect, the present disclosure further provides an electronic device, the device comprising:

[0018] one or more processors;

[0019] a storage device for storing one or more programs,

[0020] When one or more programs are executed by one or more processors, the one or more processors implement the method provided in the first aspect.

[0021] In a fifth aspect, the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the method provided in the first aspect when executed by a processor.

[0022] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0023] The present invention discloses an instrument labeling method, system, device, equipment and medium, the method comprising: obtaining an instrument image captured by a positioning camera of a target instrument, and obtaining a first perspective image obtained by a target imaging device at a first imaging position through perspective of a target object; wherein the target instrument has an instrument calibration pattern, and the target object is an object acted upon by the target instrument; performing corner point recognition on the instrument image to obtain initial corner point coordinates in a camera coordinate system of the positioning camera; determining first marking information of the target instrument on the first perspective image based on the initial corner point coordinates and a coordinate system conversion relationship; wherein the coordinate system conversion relationship is a conversion relationship between a camera coordinate system and a device coordinate system of the target imaging device; marking the target instrument on the first perspective image based on the first marking information to obtain a first marked image. In the above scheme, the target device acts on the target object and the target device is marked with a device calibration pattern. The target imaging device performs perspective on the target object to generate a first perspective image. The positioning camera performs image acquisition on the target device to obtain a device image. The first marking information is obtained by performing corner point recognition and coordinate system mapping on the device image. The target device is visually marked on the first perspective image according to the first marking information. The positioning camera and the target imaging device are organically combined in function, which not only realizes the perspective of the target object, but also realizes the position marking of the target device on the perspective image. The reliability of the marking is improved by automated marking. The user can intuitively observe the position of the target device on the marked image, so that the user can obtain corresponding image feedback when operating the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 A schematic diagram of a flow chart of a device labeling method provided in an embodiment of the present disclosure;

[0027] Figure 2 A schematic diagram of a device calibration pattern provided in an embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram of a target imaging device provided by an embodiment of the present disclosure;

[0029] Figure 4A schematic diagram of determining the distance between a positioning camera and a target imaging device provided by an embodiment of the present disclosure;

[0030] Figure 5 A schematic diagram of determining the distance from a positioning camera to the center of a field of view provided in an embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of a process for determining first identification information provided by an embodiment of the present disclosure;

[0032] Figure 7 A schematic diagram of another process for determining first identification information provided by an embodiment of the present disclosure;

[0033] Figure 8 A schematic diagram of a first marked image provided by an embodiment of the present disclosure;

[0034] Fig. 9 A schematic diagram of a process for determining a coordinate transformation relationship provided in an embodiment of the present disclosure;

[0035] Fig.10 A schematic diagram of obtaining a calibration optical image and a calibration radiographic image provided by an embodiment of the present disclosure;

[0036] Fig.11 A schematic diagram of a preset calibration plate provided in an embodiment of the present disclosure;

[0037] Fig.12 A schematic diagram of a process for determining a second marked image provided by an embodiment of the present disclosure;

[0038] Fig.13 A schematic diagram of an angle difference marking pattern provided in an embodiment of the present disclosure;

[0039] Fig.14 A schematic diagram of a second marked image provided by an embodiment of the present disclosure;

[0040] Fig.15 A schematic diagram of a device labeling method provided in an embodiment of the present disclosure;

[0041] Fig.16 A schematic diagram of the structure of an instrument labeling device provided in an embodiment of the present disclosure;

[0042] Fig.17 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0045] C-arm is an intraoperative imaging device widely used in spinal surgery to assist doctors in determining the spatial position of the instrument. In related technologies, 3D C-arm can intuitively convey the current position and depth of the instrument to the user through its own depth measurement function, but the radiation dose is relatively high.

[0046] The radiation dose of a 2D C-arm is small, but it lacks 3D depth information. Users need to build a corresponding 3D model in their minds based on 2D perspective images from multiple perspectives, and determine the position of the instrument on the object based on the 3D model. However, this method of manually determining the position of the instrument relies on the user's experience and has low reliability. How to realize the position annotation of the instrument in the corresponding view based on the 2D C-arm is an urgent problem to be solved.

[0047] In order to solve at least one of the above technical problems, the device labeling method provided by the embodiment of the present disclosure is described below. In the embodiment of the present disclosure, the device labeling method can be performed by an electronic device. The electronic device can include a device with communication functions such as a tablet computer, a desktop computer, a laptop computer, etc., and can also include a device simulated by a virtual machine or a simulator.

[0048] Figure 1 FIG. 2 is a flow chart of a device labeling method provided by an embodiment of the present disclosure. Figure 1 As shown, the device labeling method may include the following steps.

[0049] Step 101, obtain an instrument image captured by a positioning camera on a target instrument, and obtain a first perspective image obtained by a target imaging device on a target object at a first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is the object acted upon by the target instrument.

[0050] Among them, the positioning camera can be used to locate the target instrument. The positioning camera can be a camera with a three-dimensional coordinate measurement function. For example, the positioning camera can be a binocular depth camera based on stereoscopic vision, and the present embodiment does not limit the functional parameters of the binocular depth camera. For example, the frame rate of the binocular depth camera is 30 frames per second, the resolution is 1280x720, and the field of view (FOV) is 65°×40°, which is also called the field of view angle. The target instrument can be an instrument to be positioned and marked, and the present embodiment does not limit the type of the target instrument. For example, the target instrument can be a medical instrument such as a puncture needle. The present embodiment also does not limit the target object acted on by the target instrument. For example, the target object can be a surgical object such as a simulation model. Optionally, the target object can be located on a matching bed of the target imaging device. The instrument image can be an image recorded with the target instrument. The instrument image can be an image captured by the positioning camera of the target instrument when the target imaging device rotates to the first imaging position. Optionally, the instrument image can be updated in real time, thereby realizing real-time recording of user operations and real-time marking of the target instrument.

[0051] The target imaging device can be used to perform X-ray fluoroscopic imaging of the target object at multiple angles. The target imaging device can rotate around the target object and stop at different angles, which are imaging positions. X-ray imaging of the target object is performed at this angle. The present embodiment does not limit the type of the target imaging device. For example, the target imaging device can be one of a two-dimensional C-arm, a three-dimensional C-arm, and an O-arm. The first imaging position can be an angle at which the target imaging device performs fluoroscopic imaging of the target object. The present embodiment does not limit the first imaging position. For example, the first imaging position can be an anteroposterior position or a lateral position. The first fluoroscopic image can be an image obtained by the target imaging device performing fluoroscopic imaging of the target object at the first imaging position. For example, the first fluoroscopic image can include an anteroposterior image or a lateral image.

[0052] The device calibration pattern can be used to track and calibrate the target device, and the device calibration pattern can be a pattern attached to the target device. The device calibration pattern can be a pattern with known pattern features. Figure 2 A schematic diagram of a device calibration pattern provided in an embodiment of the present disclosure, such as Figure 2As shown, in some embodiments of the present disclosure, the instrument calibration pattern includes at least three unit corner point patterns, and at least three unit corner point patterns are arranged in a straight line along a guide line, and the guide line is coaxial with the axis of the target instrument. A unit corner point pattern can be a basic pattern corresponding to a corner point, and the unit corner point pattern can be an X corner point pattern. The corner point can be a point in the pattern where the grayscale changes suddenly along different directions (for example, horizontal and vertical directions). The guide line can be a straight line that guides the arrangement of the unit corner point patterns in the instrument calibration pattern, and the corner points of each unit corner point pattern in the instrument calibration pattern can all be on the guide line.

[0053] In an embodiment of the present disclosure, when the target imaging device is located at the first imaging position, the target imaging device performs perspective on the target object to obtain a first perspective image, and the target imaging device transmits the first perspective image to the instrument labeling device. The instrument labeling device receives the first perspective image. The user operates the target instrument, and the target instrument acts on the target object. When the target imaging device remains at the first imaging position, the positioning camera captures an image of the target instrument to obtain an instrument image, and the positioning camera transmits the instrument image to the instrument labeling device. The instrument labeling device receives the instrument image.

[0054] In some embodiments of the present disclosure, the positioning camera is rigidly connected to the arc-shaped arm of the target imaging device through a connecting rod, and the angle between the positioning camera and the ray emitting component in the target imaging device is a preset angle.

[0055] Among them, the connecting rod can be a rod connecting the positioning camera and the target imaging device. The arc arm is also called a cantilever, and the arc arm can be an arm with an arc-shaped structure in the target imaging device, and the arc arm can be used to carry a ray emission component, a detector component, etc. The arc arm can include a C-shaped arm in a C-arm, an O-shaped arm in an O-arm, etc. The ray emission component can be a functional component for generating X-rays in the target imaging device. The detector component can be a component in the target imaging device for receiving X-rays that pass through the target object, and the detector component is also called an image receiving component. Figure 3 A schematic diagram of a target imaging device provided by an embodiment of the present disclosure, such as Figure 3 As shown, the ray emitting assembly and the detector assembly are respectively located at two ends of the arc-shaped arm, and the positioning camera is connected to the arc-shaped arm of the target imaging device.

[0056] The preset angle may be the angle between the first line and the second line, the first line may be the connection between the ray emitting assembly and the rotation center point of the target imaging device, and the second line may be the connection between the target camera and the rotation center point. The preset angle may be half of the predicted rotation angle, and the predicted rotation angle may be the estimated angle difference between two imaging positions where the target imaging device performs perspective on the target object. For example, the imaging positions include the anteroposterior position and the lateral position, and the predicted rotation angles of the anteroposterior position and the lateral position are 90°, then the preset angle may be 45°.

[0057] In the above scheme, the angle between the positioning camera and the ray emitting component is set to a preset angle. By adjusting the preset angle, when the target imaging device is located at the imaging position, the observation angle of the positioning camera to the target instrument is not too large, thereby reducing the distortion of the collected instrument image. Without moving the target imaging device, the tracking and marking of the target instrument is achieved, thereby improving the accuracy of tracking and marking the target instrument.

[0058] In some embodiments of the present disclosure, the distance between the center of the field of view of the positioning camera and the rotation center of the target imaging device is less than a distance threshold.

[0059] The center of the field of view may be the center point of the field of view of the positioning camera. The rotation center may be the center point around which the arc arm of the target imaging device rotates. The distance threshold may be adjusted according to user needs, etc., and is not limited in this embodiment. For example, the distance threshold may be 0.

[0060] In this embodiment, the distance between the center of the field of view of the positioning camera and the rotation center of the target imaging device is adjusted to a smaller value, so that the field of view of the positioning camera can better cover the area where the target device acts on the target object. Thus, the field of view of the positioning camera can cover the user's operation.

[0061] In some embodiments of the present disclosure, the distance between the positioning camera and the target imaging device is determined according to the distance from the focus of the target imaging device to the detector in the target imaging device and the field of view parameters of the positioning camera.

[0062] The distance between the positioning camera and the target imaging device may be the distance between the positioning camera and the target imaging device along the radial direction of the arc arm. The field of view parameter may be a parameter characterizing the field of view feature, and the field of view parameter may include a field of view size and a field of view angle. The field of view size may characterize the size of the field of view, and the field of view angle may characterize the angle range observed by the positioning camera.

[0063] In this embodiment, the observation direction of the positioning camera can be along the radial direction of the arc arm. The distance between the positioning camera and the target imaging device (i.e., the length of the connecting rod) cannot be too short, otherwise, although the field of view of the positioning camera increases, the tracking accuracy will decrease. And the distance between the positioning camera and the target imaging device cannot be too long, otherwise it is difficult to completely cover the operating field of view of the target object. The ideal distance can make the field of view just completely cover the operating field of view or be slightly larger than the operating field of view. The operating field of view includes the target device, and the operating field of view can also include an angle difference marking pattern for tracking the target imaging device. The field of view size can be 600mm.

[0064] The center of the field of view of the positioning camera is defined at the rotation center of the target imaging device. The distance from the target imaging device to the rotation center can be half the distance from the target imaging device to the detector assembly. For example, if the distance from the focus of the target imaging device to the detector assembly (Source to Image Receptor Distance, SID) is 1160mm, then the distance from the focus of the target imaging device to the rotation center is 0.5SID, that is, 580mm.

[0065] Figure 4 A schematic diagram of determining the distance between a positioning camera and a target imaging device provided by an embodiment of the present disclosure, such as Figure 4 As shown, if the distance from the positioning camera to the center of the field of view is determined to be Depth, then the distance between the positioning camera and the target imaging device is: 0.5SID–Depth.

[0066] Figure 5 A schematic diagram of determining the distance from a positioning camera to the center of the field of view provided in an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the distance from the positioning camera to the center of the field of view can be determined according to the field of view size and the field of view angle. Specifically, Depth = 0.5*S / tan(FOV / 2), where Depth is the distance from the positioning camera to the center of the field of view, S is the field of view size, and FOV is the field of view angle. For example, if the field of view angle of the positioning camera in the horizontal direction is 65° and the field of view angle in the vertical direction is 40°, then the distance from the positioning camera to the center of the field of view calculated by the horizontal positioning camera is 833mm, and the distance from the positioning camera to the center of the field of view calculated by the vertical positioning camera is 470mm. It can be seen that the horizontal positioning of the camera is difficult to meet the actual needs, so the positioning camera is vertically positioned. The distance between the positioning camera and the target imaging device is: half of the distance from the focus of the target imaging device to the detector in the target imaging device minus the distance from the positioning camera to the center of the field of view. Specifically, the distance between the positioning camera and the target imaging device is 580mm-470mm=110mm.

[0067] In the above scheme, the distance between the positioning camera and the target imaging device is determined according to the distance from the focus of the target imaging device to the detector in the target imaging device and the field of view parameters of the positioning camera, so that the field of view of the positioning camera can cover the corresponding operating field of view without moving the target imaging device.

[0068] Step 102: perform corner point recognition on the instrument image to obtain initial corner point coordinates in the camera coordinate system of the positioning camera.

[0069] The camera coordinate system may be a coordinate system determined based on a positioning camera. The initial corner point coordinates may be coordinates obtained by identifying corner points in a device calibration pattern based on the device image.

[0070] In this embodiment, the device annotation apparatus can identify the corner points of the device calibration pattern in the device image according to a preset corner point recognition algorithm to obtain the corresponding initial corner point coordinates. The corner point recognition algorithm can be used to identify the position of the corner point in the image. This embodiment does not limit the corner point recognition algorithm. For example, the corner point recognition algorithm can be determined based on a neural network, or the corner point recognition algorithm can be a traditional algorithm that does not use a neural network.

[0071] Step 103, determining first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship; wherein the coordinate system conversion relationship is the conversion relationship between the camera coordinate system and the device coordinate system of the target imaging device.

[0072] Among them, the coordinate system conversion relationship can be understood as an external parameter between the target imaging device and the positioning camera, and the tracking of the target device in the camera coordinate system can be mapped to the device coordinate system through the coordinate system conversion relationship. This embodiment does not limit the method for determining the coordinate system conversion relationship. The device coordinate system can be a coordinate system determined based on the target imaging device. The first marking information can be used to mark the target device on the first perspective image, and the first marking information may include an action point and / or an action direction. Among them, the action point can represent the position where the target device operates on the target object. The action direction can represent the direction in which the target device operates on the target object. If the target device is a puncture needle, the action point can be the position of the puncture needle needle head, and the action direction can be the direction of the puncture needle.

[0073] In the disclosed embodiment, the device labeling apparatus can determine the tracking result of the target device in the camera coordinate system, and then convert the tracking result to the device coordinate system to obtain the first labeling information. Alternatively, the device labeling apparatus can determine the tracking result of the target device in the device coordinate system, and determine the first labeling information according to the tracking result.

[0074] Figure 6A schematic diagram of a process for determining first identification information provided by an embodiment of the present disclosure, such as Figure 6 As shown, in some embodiments of the present disclosure, determining the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship includes:

[0075] Step 601, according to the coordinate system conversion relationship, the initial corner point coordinates are mapped from the camera coordinate system to the device coordinate system to obtain the converted corner point coordinates.

[0076] The converted corner point coordinates may be used to record the positions of the corner points in the device calibration pattern in the device coordinate system.

[0077] In this embodiment, the device annotation apparatus can perform a spatial transformation process on the initial corner point coordinates from the camera coordinate system to the device coordinate system according to the coordinate system transformation relationship to obtain the transformed corner point coordinates.

[0078] Step 602, based on the converted corner point coordinates, according to the size parameters of the target device, determine the target action point coordinates in the device coordinate system; wherein the target action point is the point where the target device acts on the target object.

[0079] Among them, the size parameter can represent the position of the specific part of the target device that acts on the target object in the overall target device. For example, if the target device is a puncture needle, the size parameter can represent the distance between the nearest corner point and the needle tip. The nearest corner point can be the corner point closest to the needle tip among the multiple corner points of the device calibration pattern. The target action point coordinates can represent the specific position of the target device acting on the target object in the device coordinate system.

[0080] In this embodiment, the device annotation device can determine the coordinates corresponding to the nearest corner point in the converted corner point coordinates, and perform straight line fitting based on the converted corner point coordinates to obtain the guide line equation. Further, the coordinates corresponding to the nearest corner point are moved along the direction corresponding to the guide line equation according to the size parameter to obtain the target action point coordinates of the target action point in the device coordinate system.

[0081] Step 603, determining first marking information according to the target action point coordinates.

[0082] In this embodiment, the device labeling apparatus may use the target action point coordinates as the specific position of the action point, and determine the action direction of the target device according to the guide line equation, and generate first labeling information including the target action point and the guide line equation.

[0083] In the above solution, the initial corner point coordinates are first converted to the device coordinate system, and the action point and action direction are determined in the device coordinate system, thereby achieving the determination of the first marking information.

[0084] Figure 7 A schematic diagram of another process for determining first identification information provided by an embodiment of the present disclosure, such as Figure 7 As shown, in some other embodiments of the present disclosure, determining the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship includes:

[0085] Step 701, based on the initial corner point coordinates and according to the size parameters of the target device, determine the coordinates of the intermediate action point of the target action point in the camera coordinate system.

[0086] The coordinates of the intermediate action point can represent the specific position where the target device acts on the target object in the camera coordinate system. The coordinates of the intermediate action point can be understood as the position of the target action point in the camera coordinate system.

[0087] In this embodiment, the device annotation device can determine the coordinates corresponding to the nearest corner point in the initial corner point coordinates, and perform straight line fitting based on the converted corner point coordinates to obtain the guide line equation in the camera coordinate system. The coordinates corresponding to the nearest corner point are moved along the direction represented by the guide line equation in the camera coordinate system according to the size parameters to obtain the coordinates of the intermediate action point in the camera coordinate system.

[0088] Step 702, according to the coordinate system conversion relationship, the intermediate action point coordinates are mapped from the camera coordinate system to the device coordinate system to obtain the target action point coordinates of the target action point.

[0089] In this embodiment, the instrument labeling device can perform a spatial transformation process on the coordinates of the intermediate action point from the camera coordinate system to the device coordinate system according to the coordinate system conversion relationship to obtain the coordinates of the target action point.

[0090] Step 703, determining first marking information according to the target action point coordinates.

[0091] In this embodiment, the device labeling device can use the target action point coordinates as the specific position of the action point of the target device on the target object. In addition, the guide line equation in the device coordinate system is determined, and the action direction of the target device is determined according to the guide line equation. The first labeling information including the target action point and the guide line equation is generated.

[0092] In the above solution, the action point in the camera coordinate system is determined, and then the action point is converted to the device coordinate system, and the first marking information is determined based on the action point.

[0093] Step 104: Mark the target device on the first perspective image according to the first marking information to obtain a first marked image.

[0094] The first marked image may be an image obtained by annotating the function of the target device on the basis of the first perspective image. The first marked image may be understood as a navigation image of the target device and may be used to guide the user to operate the target object.

[0095] In the embodiment of the present disclosure, the instrument labeling device can acquire the instrument image in real time, and calculate the first labeling information in real time based on the instrument image, and label the target instrument on the first perspective image according to the action point and / or action direction in the first labeling information, to obtain a real-time first labeling image corresponding to the instrument image that is updated in real time. Figure 8 A schematic diagram of a first marking image provided by an embodiment of the present disclosure, such as Figure 8 As shown, in the first marking image, a black solid rectangle in the mark 801 represents the action point, and a black solid line represents the action direction.

[0096] The instrument labeling method of the embodiment of the present disclosure obtains an instrument image captured by a positioning camera of a target instrument, and obtains a first perspective image obtained by a target imaging device perspectively viewing the target object at a first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is the object acted upon by the target instrument; corner point recognition is performed on the instrument image to obtain initial corner point coordinates in a camera coordinate system of the positioning camera; first marking information of the target instrument on the first perspective image is determined based on the initial corner point coordinates and a coordinate system conversion relationship; wherein the coordinate system conversion relationship is a conversion relationship between a camera coordinate system and a device coordinate system of the target imaging device; the target instrument is marked on the first perspective image based on the first marking information to obtain a first marked image. In the above scheme, the target device acts on the target object and the target device is marked with a device calibration pattern. The target imaging device performs perspective on the target object to generate a first perspective image. The positioning camera performs image acquisition on the target device to obtain a device image. The first marking information is obtained by performing corner point recognition and coordinate system mapping on the device image. The target device is visually marked on the first perspective image according to the first marking information. The positioning camera and the target imaging device are organically combined in function, which not only realizes the perspective of the target object, but also realizes the position marking of the target device on the perspective image. The reliability of the marking is improved by automated marking. The user can intuitively observe the position of the target device on the marked image, so that the user can obtain corresponding image feedback when operating the target device.

[0097] Fig. 9 A schematic diagram of a process for determining a coordinate transformation relationship provided in an embodiment of the present disclosure is shown in FIG. Fig. 9 As shown, in some embodiments of the present disclosure, before determining the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship, the method further includes:

[0098] Step 901, obtaining a calibration optical image collected by a positioning camera on a preset calibration plate and a calibration radiographic image collected by a target imaging device on the preset calibration plate; wherein the visible light corner points and X-ray corner points of the converted calibration pattern on the preset calibration plate are the same.

[0099] Among them, the preset calibration plate can be used to calibrate the spatial conversion relationship between the device based on visible light imaging and the device based on X-ray imaging. The preset calibration plate is also called a calibration plate. The preset calibration plate can be formed by superimposing or splicing material layers of different colors and different developing materials. The calibration pattern can be a pattern on the preset calibration plate, and the calibration pattern can be a pattern with known pattern features. The calibration pattern can be formed by superimposing or splicing material layers.

[0100] The calibration optical image may be an image formed by recording the light of the visible light band of the calibration pattern. The calibration radiographic image may be an image formed after performing X-ray imaging on the calibration pattern. The visible light corner point may be a corner point set in the calibration pattern based on the visible light imaging characteristics. The X-ray corner point may be a corner point set in the calibration pattern based on the X-ray imaging characteristics. It is understandable that the visible light corner point and the X-ray corner point are both corner points pre-designed on a preset calibration plate. The visible light corner point and the X-ray corner point being the same may include the visible light corner point and the X-ray corner point coinciding, or the distance between the visible light corner point and the X-ray corner point being less than a preset distance threshold. It is understandable that due to limitations such as manufacturing processes, it is difficult to achieve complete coincidence of the visible light corner point and the X-ray corner point, and therefore, when the distance between the visible light corner point and the X-ray corner point is less than the preset distance threshold, the visible light corner point and the X-ray corner point may be approximately considered to be the same.

[0101] Fig.10 A schematic diagram of obtaining a calibration optical image and a calibration radiographic image is provided in an embodiment of the present disclosure, such as Fig.10 As shown, in this embodiment, the positioning camera and the target imaging device can be relatively stationary, and the preset calibration plate is placed in the common field of view of the positioning camera and the target imaging device. The target imaging device collects images of the preset calibration plate to generate a radiographic image. When the target imaging device does not move, the positioning camera collects images of the preset calibration plate to generate an optical image.

[0102] In some embodiments of the present disclosure, the preset calibration plate includes a radiation developing layer of a first color and a radiation non-developing layer of a second color, the radiation developing layer and the radiation non-developing layer are constructed as a calibration pattern, and the thickness of the radiation developing layer is not greater than a thickness threshold.

[0103] Among them, the first color and the second color can be different colors, and the color difference between the first color and the second color can be greater than a preset color difference threshold. For example, the first color and the second color can be black and white, respectively. The radiation developing layer can be a planar layer made of a developing material, and the developing material can be a material that develops under X-rays. For example, the developing material can be a metal such as a stainless steel material. The radiation non-developing layer can be a planar layer made of a non-developing material, and the non-developing material can be a non-metallic material that does not develop under X-rays. The thickness threshold can be a preset maximum thickness of the radiation developing layer, and this embodiment does not limit the thickness threshold. By limiting the thickness of the radiation developing layer, the imaging edge of the radiation image near the X-ray corner point can be made clearer, so that the subsequently identified calibration radiation coordinates can be more accurate.

[0104] In this embodiment, the radiation developing layer is superimposed on the radiation non-developing layer in a hollow manner to form a calibration pattern, or the radiation developing layer and the radiation non-developing layer can be spliced ​​on the same plane layer to form a calibration pattern. Since the radiation developing layer and the radiation non-developing layer have different colors, the points where the visible light color suddenly changes along different directions are visible light corner points. Since the radiation developing layer and the radiation non-developing layer have different developing properties to X-rays, the colors of the radiation developing layer and the radiation non-developing layer are different under X-rays, and the points where the X-ray developing color suddenly changes along different directions are X-ray corner points. It can be understood that since the colors and developing properties of the two layers are different, the positions of the visible light corner points and the X-ray corner points in the calibration pattern are the same.

[0105] In an optional embodiment, the preset calibration plate includes a top layer and a bottom layer, the radiation non-developing layer of the bottom layer is a non-metallic material, the thickness can be 10 mm, the radiation non-developing layer does not develop under X-rays, and the color can be pure white. The radiation developing layer of the top layer can be stainless steel 316L, the thickness can be 0.1 mm, and the color can be pure black. By using high-density metal, the radiation developing layer can be developed under X-rays. The radiation developing layer is hollowed out according to a checkerboard pattern. The radiation developing layer is bonded to the radiation non-developing layer to obtain a preset calibration plate, the surface of the preset calibration plate is flat, and it cannot fall off after high-temperature sterilization. Optionally, the color of the above-mentioned radiation non-developing layer can be pure black, and the color of the radiation non-developing layer can be pure white.

[0106] Fig.11 A schematic diagram of a preset calibration plate provided in an embodiment of the present disclosure, such as Fig.11 As shown, the calibration pattern of the preset calibration plate is a checkerboard pattern. Since the visible light corner points and X-ray corner points of the calibration pattern on the preset calibration plate are the same, if the positioning camera and the target imaging device are located at the same position, the image features of the image captured for the calibration pattern are theoretically the same.

[0107] Step 902 , performing corner point detection on the calibration optical image to obtain calibration optical coordinates in the camera coordinate system, and performing corner point detection on the calibration ray image to obtain calibration ray coordinates in the device coordinate system.

[0108] The calibration optical coordinates may be two-dimensional coordinates of visible light corner points recorded in the calibration optical image, and the calibration ray coordinates may be two-dimensional coordinates of X-ray corner points recorded in the calibration ray image.

[0109] In the disclosed embodiment, the device annotation apparatus can perform corner point recognition on the calibration optical image by using a preset corner point recognition algorithm to obtain the calibration optical coordinates in the camera coordinate system. Also, the device annotation apparatus can perform corner point recognition on the calibration radiographic image by using a preset corner point recognition algorithm to obtain multiple calibration radiographic coordinates.

[0110] Step 903, determining the coordinate system conversion relationship according to the calibration optical coordinates, the calibration ray coordinates, and the standard coordinates of the converted calibration pattern.

[0111] In the embodiment of the present disclosure, the instrument labeling device can obtain the internal parameters of the positioning camera, the internal parameters of the target imaging device, the calibrated optical coordinates, the calibrated ray coordinates, and the standard coordinates of the corner points, and calculate the spatial transformation between the camera coordinate system and the ray coordinate system in a manner that optimizes the reprojection error based on the above parameters to obtain the coordinate system conversion relationship.

[0112] In some embodiments of the present disclosure, determining a target conversion relationship between a camera coordinate system and a ray coordinate system according to calibration optical coordinates, calibration ray coordinates, and standard coordinates of corner points of a calibration pattern includes:

[0113] According to the calibration optical coordinates, the internal parameters of the positioning camera, and the standard coordinates of the corner points, the first conversion relationship between the camera coordinate system and the calibration coordinate system where the calibration pattern is located is determined. According to the calibration ray coordinates, the internal parameters of the target imaging device, and the standard coordinates of the corner points, the second conversion relationship between the calibration coordinate system and the device coordinate system is determined. The relationship conversion is performed based on the first conversion relationship and the second conversion relationship to determine the target conversion relationship.

[0114] In the above scheme, the positioning camera and the target imaging device respectively acquire images of the preset calibration plate, and the coordinate system conversion relationship is determined based on the acquired images, thereby realizing the determination of the coordinate system conversion relationship at a low hardware cost. In addition, the coordinate system conversion relationship, the internal parameters of the positioning camera, and the internal parameters of the target imaging device can be determined before operating the target object, and the coordinate system conversion relationship determined once can be used multiple times.

[0115] Fig.12 A schematic diagram of a process for determining a second marked image provided by an embodiment of the present disclosure, such as Fig.12As shown, the device labeling method also includes:

[0116] Step 1201 , obtaining a second perspective image obtained by a target imaging device at a second imaging position on a target object, and an imaging angle difference between a first imaging position and a second imaging position.

[0117] The second imaging position may be another angle at which the target imaging device performs perspective imaging of the target object, and the first imaging position and the second imaging position are different. For example, the first imaging position may be a lateral position, and the second imaging position may be an anterior-posterior position. The second perspective image may be an image obtained by the target imaging device performing perspective imaging of the target object at the second imaging position. The imaging angle difference may be the angle value that the target imaging device passes through when rotating from the first imaging position to the second imaging position, and the imaging angle difference may characterize the difference in posture of the target imaging device between the first imaging position and the second imaging position.

[0118] In this embodiment, after the target imaging device completes perspective imaging of the target object at the first imaging position to obtain a first perspective image, the target imaging device rotates to the second imaging position, performs perspective imaging of the target object at the second imaging position to obtain a second perspective image, and sends the second perspective image to the instrument labeling device. The instrument labeling device receives the second perspective imaging and obtains the imaging angle difference.

[0119] In this embodiment, there are multiple methods for determining the imaging angle difference, which is not limited in this embodiment. For example, the imaging angle difference may be determined by an external angle difference determination device, or the imaging angle difference may be determined by an instrument marking device.

[0120] Taking the imaging angle difference determined by the instrument marking device as an example, the instrument marking method may further include:

[0121] Step a1, obtaining a first angle difference image and a second angle difference image captured by a positioning camera for an angle difference mark pattern, wherein the angle difference mark pattern is located in a rotation center area of ​​a target imaging device, the first angle difference image is obtained when the target imaging device rotates to a first imaging position, and the second angle difference image is obtained when the target imaging device rotates to a second imaging position.

[0122] The angle difference marking pattern can be used to calibrate the posture of the target imaging device, and the angle difference marking pattern can be a known two-dimensional pattern, which is also called an object marking pattern. This embodiment does not limit the style of the angle difference marking pattern. Fig.13 A schematic diagram of an angle difference marking pattern provided by an embodiment of the present disclosure, such as Fig.13As shown, the angle difference mark pattern may include a plurality of X corner points arranged in a surrounding manner, and the number of X corner points in the angle difference mark pattern may be recorded in a configuration file, and the number may be no less than 4. The first angle difference image may be an image captured by a positioning camera of the angle difference mark pattern when the target imaging device is rotated to a first imaging position. The second angle difference image may be an image captured by a positioning camera of the angle difference mark pattern when the target imaging device is rotated to a second imaging position.

[0123] The rotation center area may be the area around which the target imaging device rotates, and the rotation center area may be an area including the rotation center point of the target imaging device. This embodiment does not limit the rotation center area. For example, the rotation center area may be a polygonal area centered on the rotation center point, or the rotation center area may be an area within a preset height above the matching bed of the target imaging device.

[0124] In the disclosed embodiment, the target object is located on the matching bed of the target imaging device, the angle difference marking pattern can be placed on the target object, and the angle difference marking pattern is located within the field of view of the positioning camera. The target imaging device rotates to the first imaging position, the target imaging device performs perspective on the target object to obtain a first perspective image, and the target imaging device transmits the first perspective image to the instrument labeling device. The positioning camera performs image acquisition on the angle difference marking pattern to obtain a first angle difference image, and the positioning camera transmits the first angle difference image to the instrument labeling device.

[0125] After completing the image acquisition at the first imaging position, the target imaging device rotates to the second imaging position, and the target imaging device performs perspective on the target object to obtain a second perspective image, and the target imaging device transmits the second perspective image to the instrument labeling device. The positioning camera performs image acquisition on the angle difference marking pattern to obtain a second angle difference image. By acquiring the first angle difference image and the second angle difference image, the positioning camera tracks the angle difference marking pattern. Furthermore, the positioning camera transmits the first angle difference image and the second angle difference image to the instrument labeling device. The instrument labeling device receives the first angle difference image and the second angle difference image.

[0126] Step a2, performing three-dimensional corner point detection on the first angle difference image to obtain first three-dimensional coordinates, and performing corner point detection on the second angle difference image to obtain second three-dimensional coordinates;

[0127] In the disclosed embodiment, the device labeling apparatus can identify corner points of the first angle difference image, and convert the identified coordinates from two dimensions to three dimensions to obtain first three-dimensional coordinates. The device labeling apparatus can identify corner points of the second angle difference image, and convert the identified coordinates from two dimensions to three dimensions to obtain second three-dimensional coordinates.

[0128] Step a3: determining an imaging angle difference between the first imaging position and the second imaging position according to the first three-dimensional coordinates and the second three-dimensional coordinates.

[0129] In an embodiment of the present disclosure, the instrument labeling device can pair the first three-dimensional coordinates and the second three-dimensional coordinates recorded for the same corner point at the first imaging position and the second imaging position respectively, and calculate the rotation angle difference in the three-dimensional space based on the paired first three-dimensional coordinates and the second three-dimensional coordinates to obtain the imaging angle difference.

[0130] In the above scheme, the imaging angle difference of the target imaging device is determined based on the positioning camera and the angle difference marking pattern, thereby avoiding the additional setting of an external tracker independent of the target imaging device, avoiding the introduction of the space corresponding to the tracker, reducing the number of spatial transformations, reducing the consumption of computing resources, and reducing the calculation time.

[0131] Step 1202 : Mapping the first marking information from the first perspective image to the second perspective image according to the imaging angle difference to obtain second marking information.

[0132] The second marking information may be used to mark the target device on the second perspective image, and the second marking information may include an action point and / or an action direction.

[0133] In this embodiment, the instrument marking device can acquire the instrument image in real time, calculate the first marking information in real time according to the instrument image, and perform position conversion on the first marking information according to the imaging angle difference to obtain the second marking information.

[0134] Step 1203: Mark the target device on the second perspective image according to the second marking information to obtain a second marked image.

[0135] Among them, the first marked image can be an image obtained by annotating the function status of the target device based on the second perspective image. The second marked image and the first marked image can form a multi-perspective navigation image of the target device at different imaging positions, thereby guiding the user to operate the target object at multiple perspectives.

[0136] In the embodiment of the present disclosure, the device labeling apparatus can label the target device on the second perspective image according to the action point and / or action direction in the second labeling information to obtain a second labeled image that is updated in real time. Fig.14 A schematic diagram of a second marked image provided by an embodiment of the present disclosure, such as Fig.14 As shown, in the mark 1401, the white solid rectangle represents the action point, and the white solid line represents the action direction.

[0137] In the above scheme, the target device is marked in the perspective images at two imaging positions, so that the user can understand the effect of the target device on the target object in multiple image dimensions, and accurately track the target device with less radiation to the target object.

[0138] In some embodiments of the present disclosure, the instrument image is updated in real time, and the instrument labeling method further includes: in a preset view interface, displaying the instrument image and the first marked image and the second marked image corresponding to the instrument image in real time by region; wherein the first marked image and the second marked image are arranged horizontally.

[0139] The preset view interface may be an interface for real-time navigation of the target device.

[0140] In this embodiment, the instrument annotation image collects the instrument image of the target instrument in real time, and sends the instrument image to the instrument annotation device in real time. The instrument annotation device receives the instrument image in real time and determines the first perspective image and the second perspective image based on the instrument image. The instrument annotation device can display the first perspective image and the second perspective image determined based on the same instrument image in a left-right arrangement in a preset view interface in different regions. Optionally, the instrument annotation device can display the instrument image in real time below the first marked image.

[0141] In the above scheme, the target imaging device performs perspective imaging of the target object at different imaging positions, and the instrument labeling device simultaneously maps the spatial position of the target instrument into two-dimensional perspective images of multiple imaging positions, thereby realizing real-time display of the positional relationship of the target instrument relative to the target object at different viewing angles, so that the user can accurately understand the specific position of the target instrument during the operation of the target object.

[0142] The disclosed embodiment also provides an instrument annotation system, which includes a computer, a positioning camera, a target instrument, and a target imaging device;

[0143] The positioning camera is used to capture the image of the target device to obtain the device image, and transmit the device image to the computer; the target imaging device is used to perform perspective on the target object at a first imaging position to obtain a first perspective image, and transmit the first perspective image to the computer; the computer is used to implement the above-mentioned device labeling method based on the device image and the first perspective image.

[0144] The computer may be understood as a computer for navigating the target device. The present embodiment does not limit the type of the computer. For example, the computer may be a laptop or a desktop computer. The present embodiment does not limit the configuration of the computer. For example, the computer may be configured with a graphics processing unit (GPU) and have at least 6G of memory.

[0145] Next, the instrument labeling method in the embodiment of the present disclosure is further explained through a specific example. Fig.15 A schematic diagram of an instrument labeling method provided in an embodiment of the present disclosure, in which the target imaging device is a C-arm, such as Fig.15 As shown, in the device labeling method, the target device and the target imaging device are tracked based on stereo vision, and the device labeling method includes:

[0146] Perform positioning camera calibration and C-arm calibration. Specifically, the positioning camera calibration and C-arm calibration can be completed before using the target instrument to operate the target object, and only need to be performed once. The positioning camera is calibrated through the video stream captured by the positioning camera to obtain the internal parameters of the positioning camera; the C-arm is calibrated through the perspective image captured by the C-arm to obtain the internal parameters of the C-arm. Based on the internal parameters, a high-precision spatial posture can be obtained, and the internal parameters may include: at least one of the focal length, the projection center, and the deformation coefficient, wherein the internal parameters of the positioning camera are used for posture tracking based on binocular vision.

[0147] Furthermore, the positioning camera and the C-arm are calibrated together. Specifically, after determining the internal parameters, the instrument annotation device can calculate the spatial transformation of the camera coordinate system of the positioning camera to the projection center coordinate system of the C-arm according to the internal parameters of the positioning camera and the internal parameters of the C-arm through stereo calibration, and obtain the coordinate system transformation relationship, which is the external parameter, and the external parameter can be used to map the position in the camera coordinate system to the device coordinate system where the C-arm is located.

[0148] During the process of the target instrument operating the target object, the instrument annotation device can track the spatial position of the target instrument in real time, and project the spatial position of the target instrument onto the first perspective image and the second perspective image based on the internal and external parameters obtained by calibration, thereby realizing dual-view guided operation.

[0149] like Fig.15 As shown, the process of realizing the dual-view guidance includes: estimating the posture of the target device by positioning the camera intrinsic parameters and the video stream, and estimating the posture of the C-arm by positioning the camera intrinsic parameters and the video stream, and realizing dual-view navigation based on the posture estimation result of the target device, the posture estimation result of the C-arm and the external parameters.

[0150] Specifically, the C-arm is located at a first imaging position, and the angle difference marking pattern is placed within the tracking field of view of the positioning camera. After the C-arm performs perspective on the target object, the C-arm transmits the first perspective image to the instrument labeling device. The instrument labeling device acquires the first perspective image of the first imaging position, and acquires the first angle difference image of the angle difference marking pattern acquired by the positioning camera, and determines the first three-dimensional coordinates based on the first angle difference image. After completing the image acquisition at the first imaging position, the C-arm moves to the second imaging position, and after the C-arm performs perspective on the target object, the C-arm transmits the second perspective image to the instrument labeling device. The instrument labeling device acquires the second perspective image of the second imaging position, and acquires the second angle difference image of the angle difference marking pattern acquired by the positioning camera, and determines the second three-dimensional coordinates based on the second angle difference image.

[0151] Furthermore, the first perspective image recorded at the first imaging position and the second perspective image recorded at the second imaging position are respectively displayed in two views arranged in parallel in the preset view interface. The posture of the C-arm is estimated based on the first three-dimensional image and the second three-dimensional image to obtain the imaging angle difference. The target device is tracked by the positioning camera, and the spatial position of the target device is simultaneously mapped to the first perspective image and the second perspective image through the imaging angle difference to obtain the first marked image and the second marked image. The preset view interface also includes a view that displays the left and right video streams of the positioning camera, and in this view, the corner points that can be tracked in the device image are displayed in a preset style, and the preset style can be a blue border.

[0152] In the above scheme, the target object is imaged at different angles by the target imaging device, and the rotation of the target imaging device is tracked by the angle difference marking pattern, so that the position relationship of the target device relative to the target object can be displayed in real time at different viewing angles, thereby guiding the user to operate the target object.

[0153] Moreover, by fixing the positioning camera on the target imaging device, the target imaging device can not only see through human tissue, but also use stereoscopic vision for spatial positioning. One camera can track the target device at multiple imaging positions, and the setting of the camera position takes into account both the field of view and the tracking accuracy. By navigating on multiple two-dimensional images, the user can operate the target object without three-dimensional navigation. In addition, the data relied on by the navigation comes from the real-time imaging of the target imaging device at multiple perspectives during the operation of the target object. The navigation data is reliable, and the reliability of the navigation results is also strong.

[0154] Fig.16 A schematic structural diagram of an instrument labeling device provided in an embodiment of the present disclosure is shown.

[0155] like Fig.16As shown, the device annotation device 1600 may include:

[0156] The first acquisition module 1601 is used to acquire an instrument image captured by the positioning camera on the target instrument, and acquire a first perspective image obtained by the target imaging device perspectively viewing the target object at the first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is the object acted upon by the target instrument;

[0157] The recognition module 1602 is used to perform corner point recognition on the device image to obtain initial corner point coordinates in the camera coordinate system of the positioning cameras;

[0158] A conversion module 1603 is used to determine the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship; wherein the coordinate system conversion relationship is the conversion relationship between the camera coordinate system and the device coordinate system of the target imaging device;

[0159] The first marking module 1604 is configured to mark the target device on the first perspective image according to the first marking information to obtain a first marked image.

[0160] Optionally, the positioning camera is rigidly connected to the arc-shaped arm of the target imaging device through a connecting rod, and the angle between the positioning camera and the ray emitting component in the target imaging device is a preset angle.

[0161] The distance between the center of the field of view of the positioning camera and the rotation center of the target imaging device is less than a distance threshold,

[0162] The distance between the positioning camera and the target imaging device is determined according to the distance from the focus of the target imaging device to the detector in the target imaging device and the field of view parameter of the positioning camera.

[0163] Optionally, the conversion module 1603 is used to:

[0164] According to the coordinate system conversion relationship, the initial corner point coordinates are mapped from the camera coordinate system to the device coordinate system to obtain the converted corner point coordinates;

[0165] On the basis of the conversion corner point coordinates, according to the size parameters of the target device, the target action point coordinates of the target action point in the device coordinate system are determined; wherein the target action point is the point where the target device acts on the target object;

[0166] The first marking information is determined according to the target action point coordinates.

[0167] Optionally, the conversion module 1603 is used to:

[0168] Based on the coordinates of the initial corner points, according to the size parameters of the target device, determine the coordinates of the intermediate points of the target action points in the camera coordinate system;

[0169] According to the coordinate system conversion relationship, the coordinates of the intermediate action point are mapped from the camera coordinate system to the device coordinate system to obtain the target action point coordinates of the target action point;

[0170] The first marking information is determined according to the target action point coordinates.

[0171] Optionally, the device labeling apparatus 1600 further includes:

[0172] A second acquisition module is used to acquire a calibration optical image collected by the positioning camera from a preset calibration plate and a calibration radiographic image collected by the target imaging device from the preset calibration plate before determining the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship; wherein the visible light corner points and X-ray corner points of the converted calibration pattern on the preset calibration plate are the same;

[0173] A detection module, configured to perform corner point detection on the calibration optical image to obtain calibration optical coordinates in the camera coordinate system, and to perform corner point detection on the calibration ray image to obtain calibration ray coordinates in the device coordinate system;

[0174] A determination module is used to determine the coordinate system conversion relationship according to the calibration optical coordinates, the calibration ray coordinates, and the standard coordinates of the converted calibration pattern.

[0175] Optionally, the device labeling apparatus 1600 further includes:

[0176] A third acquisition module is used to acquire a second perspective image obtained by the target imaging device at a second imaging position on the target object, and an imaging angle difference between the first imaging position and the second imaging position;

[0177] a mapping module, configured to map the first marking information from the first perspective image to the second perspective image according to the imaging angle difference, so as to obtain second marking information;

[0178] The second marking module is used to mark the target device on the second perspective image according to the second marking information to obtain a second marked image.

[0179] It should be noted that Fig.16 The illustrated instrument labeling apparatus 1600 can execute each step in the above-mentioned instrument labeling method embodiment, and realize each process and effect in the above-mentioned instrument labeling method embodiment, which will not be described in detail here.

[0180] Fig.17 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0181] like Fig.17 As shown, the electronic device may include a processor 1701 and a memory 1702 storing computer program instructions.

[0182] Specifically, the processor 1701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0183] Memory 1702 may include a large capacity memory for information or instructions. By way of example and not limitation, memory 1702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, memory 1702 may include removable or non-removable (or fixed) media. Where appropriate, memory 1702 may be inside or outside the integrated gateway device. In a particular embodiment, memory 1702 is a non-volatile solid-state memory. In a particular embodiment, memory 1702 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically alterable ROM (EAROM) or flash memory, or a combination of two or more of these.

[0184] The processor 1701 reads and executes the computer program instructions stored in the memory 1702 to perform the steps of the instrument labeling method provided in the embodiment of the present disclosure.

[0185] In one example, the electronic device may further include a transceiver 1703 and a bus 1704. Fig.17 As shown, the processor 1701, the memory 1702 and the transceiver 1703 are connected via a bus 1704 and communicate with each other.

[0186] The bus 1704 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 1704 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0187] The following is an embodiment of a computer-readable storage medium provided in an embodiment of the present disclosure. The computer-readable storage medium and the instrument labeling method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, reference can be made to the embodiment of the above-mentioned instrument labeling method.

[0188] This embodiment provides a storage medium containing computer executable instructions, which are used to perform an instrument labeling method when executed by a computer processor.

[0189] Of course, the storage medium containing computer executable instructions provided by the embodiment of the present disclosure is not limited to the above method operations, and the computer executable instructions can also execute related operations in the instrument labeling method provided by any embodiment of the present disclosure.

[0190] Through the above description of the implementation method, the technical personnel in the relevant field can clearly understand that the present disclosure can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions to enable a computer cloud platform (which can be a personal computer, server, or network cloud platform, etc.) to execute the instrument labeling method provided by each embodiment of the present disclosure.

[0191] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A device labeling method, characterized in that: include: Acquire an instrument image captured by a positioning camera of a target instrument, and acquire a first perspective image obtained by a target imaging device perspectively observing a target object at a first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is the object acted upon by the target instrument; Performing corner point recognition on the device image to obtain initial corner point coordinates in the camera coordinate system of the positioning camera; Determine the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship; wherein the coordinate system conversion relationship is the conversion relationship between the camera coordinate system and the device coordinate system of the target imaging device; The target device is marked on the first perspective image according to the first marking information to obtain a first marked image.

2. The method according to claim 1, characterized in that The positioning camera is rigidly connected to the arc-shaped arm of the target imaging device through a connecting rod, and the angle between the positioning camera and the ray emitting component in the target imaging device is a preset angle. The distance between the center of the field of view of the positioning camera and the rotation center of the target imaging device is less than a distance threshold, The distance between the positioning camera and the target imaging device is determined according to the distance from the focus of the target imaging device to the detector in the target imaging device and the field of view parameter of the positioning camera.

3. The method according to claim 1, characterized in that Determining first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship includes: According to the coordinate system conversion relationship, the initial corner point coordinates are mapped from the camera coordinate system to the device coordinate system to obtain the converted corner point coordinates; On the basis of the conversion corner point coordinates, according to the size parameters of the target device, the target action point coordinates of the target action point in the device coordinate system are determined; wherein the target action point is the point where the target device acts on the target object; The first marking information is determined according to the target action point coordinates.

4. The method according to claim 1, characterized in that: Determining first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship includes: Based on the coordinates of the initial corner points, according to the size parameters of the target device, determine the coordinates of the intermediate points of the target action points in the camera coordinate system; According to the coordinate system conversion relationship, the coordinates of the intermediate action point are mapped from the camera coordinate system to the device coordinate system to obtain the target action point coordinates of the target action point; The first marking information is determined according to the target action point coordinates.

5. The method according to claim 1, characterized in that Before determining the first marking information of the target device on the first perspective image according to the initial corner point coordinates and the coordinate system conversion relationship, the method further includes: Acquire a calibration optical image collected by the positioning camera on a preset calibration plate and a calibration radiographic image collected by the target imaging device on the preset calibration plate; wherein the visible light corner points and X-ray corner points of the converted calibration pattern on the preset calibration plate are the same; Performing corner point detection on the calibration optical image to obtain calibration optical coordinates in the camera coordinate system, and performing corner point detection on the calibration ray image to obtain calibration ray coordinates in the device coordinate system; The coordinate system conversion relationship is determined according to the calibration optical coordinates, the calibration ray coordinates, and the standard coordinates of the converted calibration pattern.

6. The method according to claim 1, characterized in that The method further comprises: Acquire a second perspective image obtained by the target imaging device at a second imaging position on the target object, and an imaging angle difference between the first imaging position and the second imaging position; Mapping the first marking information from the first perspective image to the second perspective image according to the imaging angle difference to obtain second marking information; The target device is marked on the second perspective image according to the second marking information to obtain a second marked image.

7. A device labeling system, characterized in that: Includes computers, positioning cameras, target instruments, and target imaging equipment; The positioning camera is used to acquire an image of the target device to obtain a device image, and transmit the device image to the computer; the target imaging device is used to perform perspective imaging on the target object at a first imaging position to obtain a first perspective image, and transmit the first perspective image to the computer; The computer is configured to implement the method of any one of claims 1 to 6 based on the instrument image and the first perspective image.

8. An instrument marking device, characterized in that: include: A first acquisition module is used to acquire an instrument image captured by a positioning camera on a target instrument, and to acquire a first perspective image obtained by a target imaging device perspectively viewing a target object at a first imaging position; wherein the target instrument has an instrument calibration pattern, and the target object is an object acted upon by the target instrument; A recognition module, used for performing corner point recognition on the device image to obtain initial corner point coordinates in the camera coordinate system of the positioning cameras; A conversion module, used to determine the first marking information of the target device on the first perspective image according to the initial corner point coordinates and a coordinate system conversion relationship; wherein the coordinate system conversion relationship is a conversion relationship between the camera coordinate system and the device coordinate system of the target imaging device; The first marking module is used to mark the target device on the first perspective image according to the first marking information to obtain a first marked image.

9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 6.

Images