Spatial pose pointing and positioning method for surgical field video acquisition

Abstract

The invention discloses a spatial pose pointing and positioning method for surgical field video acquisition, and belongs to the technical field of clinical medical teaching instruments. The method comprises the following steps: designing a two-dimensional code target for indicating the pose of a shooting target, and obtaining a coordinate system transformation model between each two-dimensional code on the target and a target body; after receiving an identification instruction sent by the two-dimensional code target, the surgical field video acquisition equipment controls a camera to shoot the two-dimensional code target, identifies the pose of a two-dimensional code pattern, and determines the spatial pose of the target according to the coordinate system transformation model after error compensation; determining an expected shooting pose of the camera according to the calculated target space pose, and controlling the camera to move to the expected shooting pose to shoot the shooting target; visual servo control is adopted to adjust the shooting pose of the camera. The efficiency of surgical operation teaching video recording is improved, resource waste is reduced, and a user can autonomously indicate and adjust the shooting pose of the camera.

Description

Technical Field

[0001] The present invention belongs to the technical field of clinical medical teaching instruments, and specifically refers to a spatial posture pointing positioning method for surgical field video acquisition. Background Art

[0002] When recording surgical teaching videos in surgical operating rooms, in order to ensure that the camera can capture high-quality videos, it is often necessary to adjust the camera's position and align the camera's optical axis with the target. In this way, the captured video can ensure that the target is accurately located in the center of the video image, and the information on the edge of the target will not be lost. In addition, keeping the target in the center of the image can also improve the quality of the video and provide viewers with a better viewing experience.

[0003] However, in the actual surgical field video recording scene, the surgeon and the surrounding medical staff are all moving, which may block the camera shooting the surgical field video. Therefore, when recording teaching videos of surgical operations, it is often necessary to equip a special photographer to hold or shoulder the camera to adjust the shooting angle in real time. However, some surgical operations last for a long time, and the photographer holding or shouldering the camera for a long time will cause great physical exertion. In addition, the space in the operating room is often limited, and the photographer shooting in the operating room will also cause inconvenience to the work of medical staff.

[0004] In response to the above problems, some highly automated surgical field video acquisition devices have gradually emerged and have begun to be widely used in surgical teaching video recording. Although this type of surgical field video acquisition device does not require a professional photographer to adjust the shooting angle, it usually requires additional nurses to operate it. This results in that in actual use, the surgical field video is not recorded completely according to the surgeon's surgical ideas, and the surgical teaching video may lose some important details. Summary of the invention

[0005] At present, when recording surgical teaching videos in the operating room, in order to ensure the quality of the video, a special operator is usually required to adjust the shooting posture of the camera in the surgical field video acquisition device, which causes a certain degree of resource waste and brings inconvenience to the work of medical staff in the operating room. In the actual surgical field video shooting process, in order to record a more suitable video, the surgeon needs to remotely adjust the camera shooting posture. For such an application scenario, the present invention provides a spatial posture pointing and positioning method for surgical field video acquisition, the purpose of which is to improve the efficiency of the surgical field video acquisition device recording, reduce resource waste, and enable the surgeon to independently instruct to adjust the shooting posture of the camera in the surgical field video acquisition device.

[0006] The spatial posture pointing positioning method for surgical field video acquisition provided by the present invention comprises the following steps:

[0007] Step 1: Design a QR code target for indicating the position of the target to be photographed, and obtain the coordinate system transformation model between each QR code coordinate system on the target and the target body coordinate system; the QR code target has a QR code shell in the shape of a prism, and a QR code pattern is drawn on the top surface and four sides of the prism. Let the target body coordinate system be G A , the coordinate system of the QR code labeled i is G i The coordinate transformation matrix of the QR code coordinate system labeled i in the base coordinate system of the surgical field video acquisition device is T i , used to represent the spatial position of the QR code, G i With G A The coordinate transformation matrix between The operator holds the two-dimensional code target and points it to the desired shooting position, and controls the two-dimensional code target to send a recognition instruction to the surgical field video acquisition device.

[0008] Step 2: After receiving the recognition command from the QR code target, the surgical field video acquisition device controls the camera to shoot the QR code target, identifies the pose of the QR code pattern, and after error compensation, determines the spatial pose of the target according to the coordinate system transformation model. Pre-fit the angle error compensation model of the QR code pattern; use the QR code pose recognition algorithm to calculate the pose of each QR code pattern captured in three-dimensional space, extract the RPY angle from it, and use the angle error compensation model to correct it; calculate the new pose of the QR code image from the corrected RPY angle, and then determine the spatial pose of the target according to the coordinate system transformation model.

[0009] Step 3: Determine the desired shooting posture of the camera according to the calculated target spatial posture, and control the camera in the surgical field video acquisition device to move to the desired shooting posture to shoot the target;

[0010] Step 4: After the camera reaches the desired shooting position, use the center point of the QR code pattern on the top of the target as the feature point, and use visual servoing to adjust the shooting position of the camera in the surgical field video acquisition device so that the feature point is located at the center of the image captured by the camera. When the shooting accuracy requirement is met, stop adjusting.

[0011] In step 2, let the QR code position T of label i be i Extract the RPY angle α from i , β i and γ i , the corresponding compensation data e is obtained through the angle error compensation model αi 、e βi and e γi, calculate the corrected RPY angle: α revise =α i +e αi , β revise =β i +e βi , γ revise =γ i +e γi ; Calculate the new position T of the QR code based on the corrected RPY angle inew as follows:

[0012]

[0013] Among them, c represents the cos operation to calculate the cosine value, s represents the sin operation to calculate the sine value, and x i ,y i ,z i From T i The spatial position of the QR code extracted from .

[0014] Assume that k QR code patterns of the QR code target are currently captured, k≤5, and the RPY angle of the QR code with number i is corrected to obtain the new pose T inew , then calculate the spatial pose T of the current target in the base coordinate system of the surgical field video acquisition device Anew as follows:

[0015]

[0016] Wherein, K is the number set of the two-dimensional code pattern of the two-dimensional code target currently photographed.

[0017] In step 4, the visual servo control is used to adjust the camera shooting posture, including: first, the image captured by the camera is processed to solve the pixel coordinates (u0, v0) of the QR code feature point on the top of the target and the center of the shot image (u tar ,v tar ) pixel error; then, fuzzy proportional integral control is used to solve the motion speed (V u ,V v ); through the image Jacobian matrix, the motion speed (V u ,V v ) is mapped to the motion speed of the camera in the base coordinate system of the surgical field video acquisition device in three-dimensional space (V x ,V y ); Finally, control the camera to move until (u0,v0) and (u tar ,v tar ) is less than the threshold, and the adjustment is stopped.

[0018] The advantages and positive effects of the present invention are:

[0019] (1) For the scenario where the surgical field video acquisition device shoots a surgical teaching video, the method of the present invention does not require a dedicated operator to adjust the camera, thereby reducing the human resource cost required for shooting the scene;

[0020] (2) The method of the present invention can remotely adjust the shooting posture of the camera in the surgical field video acquisition device through the target, without the need to contact the surgical field video acquisition device, and is very applicable to some scenes with isolated shooting requirements.

[0021] (3) The method of the present invention uses multiple two-dimensional code patterns to jointly identify posture and calculates the average value as the final posture result, thereby improving the posture recognition accuracy and anti-interference ability.

[0022] (4) The method of the present invention constructs an error compensation model based on a two-dimensional code target, which can reduce the RPY angle system error of target posture recognition and further improve the accuracy of target posture recognition.

[0023] (5) The method of the present invention uses visual servoing to fine-tune the position of the camera so that the optical axis of the camera can be accurately aligned with the shooting target, so that the shooting target is located in the center of the captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is an overall flow chart of the spatial posture pointing positioning method for surgical field video acquisition of the present invention;

[0025] Figure 2 is a schematic diagram of a two-dimensional code target used in an embodiment of the present invention;

[0026] Figure 3 is a schematic diagram of determining an expected shooting posture of a camera according to an embodiment of the present invention;

[0027] Figure 4 It is a flow chart of visual servo control according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] In view of the scenario where the surgeon needs to remotely adjust the camera shooting posture in the surgical field video acquisition device during the surgical field video shooting process, the present invention provides a spatial posture pointing positioning method for surgical field video acquisition based on QR code posture recognition and visual servo control technology. The method ensures that the optical axis of the camera shooting the surgical field video can be aligned with the shooting target, so that the shooting target can be accurately located at the center of the shot image, thereby improving the quality of the shot video; while ensuring the quality of the shot video, it can also reduce the workload of the shooting personnel and improve the shooting efficiency. The method of the present invention has good applicability in the recording scene of surgical teaching videos.

[0030] In the spatial posture pointing and positioning method of the present invention, a two-dimensional code target is first used to indicate the shooting target; then the posture of the two-dimensional code is preliminarily identified, and then a pre-constructed error compensation model is used to calculate the target posture information with higher accuracy; and based on this, the shooting posture of the camera is determined, and then the camera is controlled to move to the specified posture; after the camera moves to the specified posture, it is judged according to the image captured by the camera whether the feature point of the two-dimensional code pattern on the top of the target is in the center of the shooting picture. If there is a deviation, the camera posture is continued to be adjusted through visual servoing to ensure that the target is accurately located in the center of the captured image.

[0031] like Figure 1 As shown, a spatial posture pointing positioning method for surgical field video acquisition of the present invention includes the following four steps, and the implementation of each step is described below.

[0032] Step 1: Design a QR code target to indicate the position of the target, determine the QR code pattern used for recognition, and construct a coordinate system transformation model based on the positional relationship between the QR code coordinate system and the target body coordinate system.

[0033] For the scenario of surgical operation teaching video recording, the QR code target designed in the embodiment of the present invention is as follows: Figure 2 As shown. The main structure of the QR code target includes a QR code shell, a target handle and a target control button, wherein the QR code shell is in the shape of a prism, and a total of 5 QR code labels are set on the top and side surfaces of the prism; the target handle is used by the surgeon to hold it to control the target pointing posture; the target control button is used to turn on and off the target and send target recognition instructions. The side length of the QR code pattern in the QR code target of the embodiment of the present invention is 65mm, and the mass of the target body is 265g.

[0034] When using the QR code target, the surgeon first holds the QR code target to point to the desired shooting position, and then uses the target control button to send instructions to the surgical field video acquisition device; after receiving the instructions, the surgical field video acquisition device begins to identify the target posture, and then controls the movement of the high-definition camera to shoot the desired target.

[0035] The two-dimensional code pattern used in the method of the present invention is the tag36h11 family series two-dimensional code in the AprilTag label, because this type of two-dimensional code has a strong anti-noise ability. There are 5 two-dimensional code patterns drawn on the two-dimensional code shell, namely Figure 2 There are 5 QR codes on the top surface and four sides, with IDs 0, 1, 2, 3, and 4. Let the coordinate system corresponding to each QR code be G i ,i=0,1,2,3,4. The target body coordinate system is G A , since the target shape is fixed, G i With G A The transformation relationship between them is also fixed. Let G i With G A The transformation relationship between them is

[0036] Step 2: Identify the pose of the QR code pattern, and preliminarily calculate the target's spatial pose based on the coordinate system transformation model constructed in step 1. In this process, perform error compensation on the target pose, and finally determine the target's spatial pose.

[0037] The AprilTag pose recognition algorithm and the external parameter information of the camera in the surgical field video acquisition device are used to calculate the coordinate system G of each QR code. i The pose T in three-dimensional space i , and then through the coordinate transformation relationship Calculate the target pose T corresponding to each QR code Ai , as shown in formula (1):

[0038]

[0039] Finally, T Ai The target's position T in three-dimensional space is calculated by summing and averaging. A , as shown in formula (2):

[0040]

[0041] Where k is the number of two-dimensional code patterns currently photographed by the two-dimensional code target.

[0042] After testing, it was found that when using the AprilTag pose recognition algorithm to calculate the spatial pose of the QR code, the pose angle information obtained has a certain degree of systematic error. This part of the error can be measured and statistically analyzed through preliminary experiments, and then the error compensation model can be obtained by fitting the statistical error data.

[0043] The method of the present invention measures the error data of the target two-dimensional code pattern at different RPY (roll-pitch-yaw) angles. Assume that the RPY angle of the measured two-dimensional code pattern with the number i is α i , β i and γ i , and the real RPY angle of the QR code pattern is α real , β real and γ real , based on which the RPY angle error e of the QR code pattern is calculated α 、e β and e γ , as shown in formula (3):

[0044]

[0045] Then, the error data was fitted by adaptive fuzzy neural network (ANFIS), and finally the angle error compensation model of the target QR code was obtained.

[0046] When measuring the target pose, first identify the target QR code pose T i Extract the RPY angle α from i , β i and γ i , and then pass it as input to the pre-trained error compensation model to obtain the corresponding error compensation data e αi 、e βi and e γi The corrected RPY angle can be obtained by adding this part of the error compensation data to the original data, as shown in formula (4):

[0047]

[0048] According to the corrected RPY angle, calculate the new QR code pose T inew , as shown in formula (5):

[0049]

[0050] where x i ,y i , z i T i The position vector element in is the spatial position represented by the QR code pattern; c is the abbreviation of cos, which is the operation for calculating the cosine value, and s is the abbreviation of sin, which is the operation for calculating the sine value.

[0051] Finally, referring to equations (1) and (2), we get the new target spatial pose T Anew , as shown in formula (6):

[0052]

[0053] In actual use, the number of QR codes captured each time is not necessarily five. The angle error compensation and new pose calculation are performed on the QR codes actually captured. Finally, the target pose is calculated using formula (6). The value of i is the number of the captured QR code.

[0054] Since the target is pointing to the shooting target, the shooting target is on the extension line of the target pointing direction. It is only necessary to ensure that the optical axis of the camera in the surgical field video acquisition device is aligned with the target pointing direction, so that the shooting target can be accurately located at the center of the captured image.

[0055] Step 3: According to the calculated target spatial pose, further determine the shooting pose of the camera, and then control the camera to move to the specified pose to shoot the target.

[0056] After obtaining the target spatial position T with compensated angle error Anew Afterwards, the position vector P is extracted Anew , T Anew With P Anew The relationship is shown in formula (7):

[0057]

[0058] Where R Anew is the target body coordinate system G A The rotation matrix relative to the base coordinate system of the surgical field video acquisition device. Anew Along Z Anew Direction projection to the camera's motion plane to obtain point P PTZ ,like Figure 3 As shown. Anew The direction is calculated by formula (8):

[0059]

[0060] Figure 3 The point P found in PTZ is the expected shooting position of the camera, calculated as shown in formula (9):

[0061]

[0062] When using surgical field video acquisition equipment to record surgical videos, the camera often moves in a plane at a fixed height to capture a top view of the surgical field. Therefore, the height z of the camera's desired shooting position is PTZ is known, based on which we can find the parameter w, and finally further calculate x PTZ and PTZ .

[0063] After calculating the desired shooting position of the camera, it is also necessary to determine the shooting angle of the camera. Figure 3 It can be seen that the center point P of the shooting target tar Just in Z Anew On the reverse extension line, the camera's shooting angle direction is actually Z Anew Direction, can be found through T Anew The rotation matrix R in Anew to be sure.

[0064] After calculating the desired shooting position and angle of the camera, the camera can be controlled to move to the specified shooting position and angle to shoot the desired target.

[0065] Step 4: Process the image captured by the camera, calculate the position error between the center point of the QR code on the top of the target and the center of the captured image, and judge whether the shooting accuracy requirement is met based on this. If the requirement is not met, continue to use visual servoing to fine-tune the camera posture.

[0066] After step 3 is completed, the camera has reached the specified shooting posture, but there will inevitably be errors in the whole process, such as the recognition error of the two-dimensional code, the error in posture solution and the motion error of the camera. Due to the existence of these errors, the desired target captured at this time is often not accurately located at the center of the captured image. In order to eliminate the influence of these errors, the method of the present invention uses the center point of the two-dimensional code pattern at the top of the target as the feature point, and uses visual servo control to adjust the camera shooting posture so that the center feature point of the two-dimensional code at the top of the target can be accurately located at the center of the captured image. And because the target is pointing to the shooting target, after the visual servo is completed, the target is moved away, and the actual shooting target center can also be accurately located at the center of the captured image.

[0067] The specific process of visual servo control is as follows: Figure 4 As shown in the figure, the image captured by the camera is first processed to solve the pixel coordinates U(u0,v0) of the central feature point of the QR code at the top of the target; then the pixel coordinates (u0,v0) and the center of the captured image (u tar ,v tar ) is the pixel error (Δu, Δv), as shown in formula (10):

[0068]

[0069] Then, fuzzy proportional integral control is used to solve the motion speed (V u ,V v ), as shown in formula (11):

[0070]

[0071] Where kp is the proportional coefficient and ki is the integral coefficient, which are adjusted by the fuzzy strategy; (Δu sum ,Δv sum ) is the cumulative sum of pixel errors (Δu, Δv).

[0072] Get the motion speed in the pixel coordinate system (V u ,V v ), the image Jacobian matrix is ​​used to map the motion speed of the camera in the base coordinate system of the surgical field video acquisition device in the three-dimensional space (V x ,V y ), as shown in formula (12):

[0073]

[0074] Where J is the image Jacobian matrix. Unlike the traditional image Jacobian matrix, J is just a 2×2 matrix. This is because when controlling (u0, v0) to reach the specified position (u tar ,v tar ) has only two degrees of freedom, so (V u ,V v ) is mapped to the movement speed of the camera in three-dimensional space, it only needs to be converted into the speed of the camera in the X and Y directions in the base coordinate system of the surgical field video acquisition device.

[0075] Finally, the camera is controlled to move until (u0, v0) is aligned with the center of the captured image (u tar ,v tar ) is less than the threshold, the adjustment is stopped. The specific value of the image Jacobian matrix is ​​estimated online using the Kalman filter method based on the fuzzy strategy.

[0076] All steps of the present invention are implemented by information processing by computers and other devices, and the implementation process does not limit the freedom of doctors in the diagnosis process and can be used in industry. All steps of the present invention are independent of the intervention process and do not involve traumatic or interventional treatment or treatment of living human or animal bodies.

[0077] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A spatial posture pointing positioning method for surgical field video acquisition, characterized in that: The steps include: Step 1: Design a QR code target for indicating the position of the target, determine the QR code pattern used for recognition, and build a coordinate system transformation model based on the positional relationship between the QR code coordinate system and the target body coordinate system; The two-dimensional code target has a prism-shaped shell, and a two-dimensional code pattern is drawn on the top surface and four side surfaces of the prism. The coordinate system of the target body is G A , the coordinate system of the two-dimensional code labeled i is G i , G i With G A The coordinate transformation relationship between them is The operator holds the QR code target and points it to the desired shooting position, and controls the QR code target to send a recognition instruction to the surgical field video acquisition device; Step 2: After receiving the recognition instruction sent by the QR code target, the surgical field video acquisition device controls the camera to shoot the QR code target, identifies the pose of each QR code, performs error compensation, and determines the spatial pose of the target according to the coordinate system transformation model; The angle error compensation model of the two-dimensional code pattern is pre-fitted; the pose of each captured two-dimensional code pattern in three-dimensional space is calculated using a two-dimensional code pose recognition algorithm, the RPY angle is extracted from it, and the angle error compensation model is used for correction; the new pose of the two-dimensional code is calculated from the corrected RPY angle, and then the spatial pose T of the target is determined according to the coordinate system transformation model. Anew ;RPY stands for Roll-Pitch-Yaw; Step 3: Determine the desired shooting posture of the camera according to the calculated target spatial posture, and control the camera in the surgical field video acquisition device to move to the desired shooting posture to shoot the target; Step 4: After the camera reaches the desired shooting posture, use the center point of the QR code pattern on the top of the target as the feature point, and use visual servoing to adjust the shooting posture of the camera so that the feature point is located at the center of the image captured by the camera. When the shooting accuracy requirement is met, stop adjusting.

2. The method according to claim 1, characterized in that In step 2, the position of the identified QR code with label i in the base coordinate system of the surgical field video acquisition device is assumed to be T i , from T i Extract the RPY angle α from i , β i and γ i , the corresponding compensation data e is obtained through the angle error compensation model αi 、e βi and e γi , calculate the corrected RPY angle: α revise =α i +e αi ,β revise =β i +e βi ,γ revise =γ i +e γi ; Calculate the new position T of the QR code based on the corrected RPY angle inew as follows: Among them, c represents the cos operation to calculate the cosine value, s represents the sin operation to calculate the sine value, and x i ,y i ,z i From T i The spatial position of the QR code extracted from 3. The method according to claim 1 or 2, characterized in that: In step 2, assume that k two-dimensional code patterns of the two-dimensional code target are currently photographed, k≤5, and the RPY angle of the two-dimensional code labeled i is corrected to obtain a new posture T after correction inew , then calculate the spatial pose T of the current target in the base coordinate system of the surgical field video acquisition device Anew as follows: Wherein, K is the label set of the QR code of the currently photographed QR code target.

4. The method according to claim 1, characterized in that In step 3, the target spatial position T Anew The following is obtained: Among them, R Anew is the rotation matrix of the target body coordinate system relative to the base coordinate system of the surgical field video acquisition device, P Anew is the position vector of the target body coordinate system relative to the base coordinate system of the surgical field video acquisition device; Determine the camera's shooting angle: the center point of the target is at vector Z Anew On the reverse extension line; Determine the camera's shooting position: Anew Along Z Anew The direction is projected onto the camera's motion plane to obtain the position point P PTZ , P PTZ This is the desired shooting position of the camera.

5. The method according to claim 1, characterized in that In step 4, the visual servo control is used to adjust the camera shooting posture, including: first, the image captured by the camera is processed to solve the pixel coordinates (u0, v0) of the QR code feature point on the top of the target and the center of the shot image (u tar ,v tar ) pixel error; then, fuzzy proportional integral control is used to solve the motion speed (V u ,V v ); through the image Jacobian matrix, the motion speed (V u ,V v ) is mapped to the motion speed of the camera in the base coordinate system of the surgical field video acquisition device in three-dimensional space (V x ,V y ); Finally, control the camera to move until (u0,v0) and (u tar ,v tar ) is less than the threshold, and the adjustment is stopped.

Images