Data processing method and device and trajectory capturing equipment

Abstract

The embodiment of the invention provides a data processing method and device and track capturing equipment, and the method comprises the steps: firstly obtaining magnetic data and video frame data in a motion process, and then obtaining a motion track of a motion object in the motion process according to the magnetic data and the video frame data. That is to say, the motion trail of the moving object is obtained through the magnetic data obtained through the magnetic positioning technology and the video frame data obtained through the video collection device. The method does not need to depend on experience and judgment of observers, and is not easily interfered by external factors. In addition, the method can make full use of the high-frequency sampled magnetic data of the magnetic positioning technology to dynamically predict and compensate the low-frame-rate video frame data, and can improve the positioning precision of the motion trail of the moving object compared with the method of obtaining the motion trail of the moving object by using single video frame data.

Description

Technical Field

[0001] The present application is applied to the field of signal technology, and specifically relates to a data processing method, apparatus, and trajectory capture device. Background Art

[0002] The trajectory of a moving object often needs to be captured during its motion. For example, during acupuncture, the trajectory of a needle must be captured to study the mechanism of acupuncture techniques, improve standardization of procedures, and enhance training effectiveness. However, current methods for locating the trajectory of moving objects often rely on traditional visual observation and manual judgment. This is not only limited by the observer's experience and judgment but also susceptible to interference from external factors, resulting in poor trajectory positioning accuracy. Therefore, improving the accuracy of locating the trajectory of moving objects has become a technical challenge to be solved. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method, apparatus, and trajectory capture device for improving the accuracy of locating the motion trajectory of a moving object.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0005] Acquire magnetic data and video frame data of the motion process; the magnetic data is acquired by magnetic positioning technology and is used to determine the spatial prediction trajectory of the moving object during the motion process; the video frame data is acquired by a video acquisition device and includes video frames of the moving object;

[0006] The motion trajectory of the moving object during the motion process is acquired according to the magnetic data and the video frame data.

[0007] Optionally, the method for obtaining the video frame data includes:

[0008] The video frame data is acquired through m video acquisition devices, where m is an integer greater than 2, and each of the m video acquisition devices has a different shooting angle.

[0009] Optionally, if the m video acquisition devices include a first camera and a second camera, before acquiring the first video frame data through the m video acquisition devices, the method further includes:

[0010] The first camera and the second camera are calibrated respectively to obtain intrinsic parameters of the first camera and the second camera, and to obtain a relative position and posture between the first camera and the second camera.

[0011] Optionally, the method further includes:

[0012] determining a predicted spatial trajectory of the moving object during the movement based on the magnetic data; and determining an observed spatial trajectory of the moving object during the movement based on the video frame data;

[0013] The acquiring, according to the magnetic data and the video frame data, a motion trajectory of the moving object during the motion process, includes:

[0014] The spatial prediction trajectory and the spatial observation trajectory are fused to obtain the motion trajectory of the moving object during the motion process.

[0015] Optionally, if the video frame data includes video frames at multiple moments, determining the spatial observation trajectory of the moving object during the movement based on the video frame data includes:

[0016] For each video frame at the multiple moments, execute:

[0017] Extracting feature points of a video frame; the video frame includes a first sub-video frame acquired by the first camera and a second sub-video frame acquired by the second camera;

[0018] Matching feature points of the first sub-video frame with feature points of the second sub-video frame to obtain at least one matching pair, wherein each matching pair in the at least one matching pair corresponds to the same spatial position point;

[0019] Obtaining a disparity corresponding to each matching pair in the at least one matching pair; obtaining a spatial observation position of the moving object based on the disparity corresponding to each matching pair, wherein the disparity corresponding to a first matching pair indicates a difference between horizontal coordinates of two feature points of the first matching pair;

[0020] The spatial observation trajectory is obtained based on the spatial observation position of the moving object at each moment.

[0021] Optionally, fusing the predicted spatial trajectory and the observed spatial trajectory to obtain the motion trajectory of the moving object during the motion process includes:

[0022] The spatial prediction trajectory and the spatial observation trajectory are fused through the Kalman filter algorithm to obtain the motion trajectory of the moving object in the motion process.

[0023] Optionally, if the moving object is acupuncture, the magnetic data is the magnetic field strength of the magnetic induction sensor during the movement, and obtaining the magnetic data includes:

[0024] The first magnetic data is obtained based on the micro permanent magnet at the acupuncture end and the magnetic field sensing system. The magnetic field sensing system includes n×n magnetic induction sensors, where n is an integer greater than 1. The n×n magnetic induction sensors are an n×n square array and are located in the moving area of ​​the moving object.

[0025] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0026] an acquisition unit, configured to acquire magnetic data and video frame data of a motion process; the magnetic data is acquired through magnetic positioning technology and is used to determine a predicted spatial trajectory of a moving object during the motion process; the video frame data is acquired by a video acquisition device and includes video frames of the moving object;

[0027] A determination unit is configured to obtain a motion trajectory of the moving object during the motion process based on the magnetic data and the video frame data.

[0028] Optionally, the acquisition unit includes m video acquisition devices, and the acquisition unit is specifically configured to:

[0029] The video frame data is acquired through the m video acquisition devices, where m is an integer greater than 2, and each of the m video acquisition devices has a different shooting angle.

[0030] In a third aspect, an embodiment of the present application provides a trajectory capture device, comprising: a needle, a magnetic field sensing system, a video capture device, and a processor; the magnetic field sensing system is connected to the processor, and the video capture device is connected to the processor;

[0031] The needle tip includes a micro permanent magnet;

[0032] The magnetic field sensing system includes n×n magnetic induction sensors, where n is an integer greater than 1, and the n×n magnetic induction sensors form an n×n square array;

[0033] The magnetic field sensing system is used to obtain magnetic data of the movement process based on the micro permanent magnet; the magnetic data is used to determine the spatial prediction trajectory of the micro permanent magnet during the movement process; and the magnetic data is sent to the processor;

[0034] The video acquisition device is used to obtain video frame data including the needle and send the video frame data to the processor;

[0035] The processor is used to obtain the motion trajectory of the needle during the motion process according to the magnetic data and the video frame data.

[0036] The present application provides a data processing method, apparatus and trajectory capture device. When executing the method, magnetic data and video frame data are first obtained during the motion process, and then the motion trajectory of the moving object during the motion process is obtained based on the magnetic data and video frame data. That is to say, the present application obtains the motion trajectory of the moving object through the magnetic data obtained by magnetic positioning technology and the video frame data obtained by the video acquisition device. The present application does not need to rely on the experience and judgment of the observer and is not easily interfered by external factors. In addition, the present application can make full use of the high-frequency sampled magnetic data of the magnetic positioning technology to dynamically predict and compensate for the low-frame-rate video frame data. Compared with using a single video frame data to obtain the motion trajectory of the moving object, the positioning accuracy of the motion trajectory of the moving object can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic structural diagram of a trajectory capture device provided in an embodiment of the present application;

[0038] Figure 2 A schematic structural diagram of a needle 101 provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the deployment of the magnetic field sensing system 102;

[0040] Figure 4 A schematic structural diagram of a dual-view video acquisition device provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a dipole model provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of obtaining a spatial observation position of a needle provided in an embodiment of the present application;

[0043] Figure 7 A flow chart of a data processing method provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of obtaining a motion trajectory provided in an embodiment of the present application;

[0045] Figure 9 A specific implementation diagram of another data processing method provided in an embodiment of the present application;

[0046] Figure 10 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] As mentioned above, the current motion trajectory of moving objects often relies on traditional visual observation and manual judgment, which is limited by the observer's experience and judgment, and is easily interfered by external factors, resulting in poor positioning accuracy of the motion trajectory.

[0048] In one implementation, the motion trajectory of the moving object can be obtained by acquiring video frame data of the moving object. For example, a marking point (such as a reflective marker or a QR code) is attached to the handle of an acupuncture needle, or an active light-emitting marker (such as an LED) is installed, and then a video capture device is used to capture a video of the acupuncture operation. A computer vision algorithm is then used to analyze the marker position frame by frame to reconstruct the three-dimensional motion trajectory of the needle. However, due to the low frame rate of the video frames captured by the video capture device, the low frame rate will result in a large time interval between the video frame data. For a fast-moving needle, some key motion states may be missed, resulting in obvious jumps and discontinuities in the reconstructed three-dimensional motion trajectory, which cannot accurately reflect the actual motion process of the needle. In other words, the motion trajectory obtained in this way has low accuracy.

[0049] In view of this, an embodiment of the present application provides a data processing method, which obtains the motion trajectory of a moving object through magnetic data obtained by magnetic positioning technology and video frame data obtained by a video acquisition device. The present application does not need to rely on the experience and judgment of the observer and is not easily interfered with by external factors. In addition, the present application can make full use of the high-frequency sampled magnetic data of the magnetic positioning technology to dynamically predict and compensate for the low-frame-rate video frame data. Compared with using a single video frame data to obtain the motion trajectory of the moving object, the positioning accuracy of the motion trajectory of the moving object can be improved.

[0050] In order to better illustrate the data processing method provided in the embodiment of the present application, a schematic description is given below by taking the moving object as a needle and the moving process as the acupuncture process of the needle as an example.

[0051] First, the application scenarios of the embodiments of the present application are introduced.

[0052] For example, the Figure 1 This is a schematic diagram of the structure of a trajectory capture device provided in an embodiment of the present application. This trajectory capture device 100 is used to implement the data processing method provided in an embodiment of the present application. This trajectory capture device 100 includes a needle 101, a magnetic field sensing system 102, a video capture device 103, and a processor 104. Processor 104 is connected to both the magnetic field sensing system 102 and the video capture device 103.

[0053] In the embodiments of this application, Figure 2This is a schematic diagram of the structure of a needle 101 provided in an embodiment of the present application. The needle 101 includes a needle body 101-1, and a micro-permanent magnet 101-2 at the end of the needle body 101-1 (i.e., the needle tip). The micro-permanent magnet 101-2 is aligned axially with the needle body 101-1, thereby forming a stable magnetic source.

[0054] The micro permanent magnet 101-2 can be cylindrical or other shapes, which is not specifically limited in the present embodiment. The micro permanent magnet 101-2 can be a neodymium magnet or a magnet made of other materials, which is not specifically limited in the present embodiment. The size of the micro permanent magnet 101-2 can be adjusted as needed, for example, the size is 1×1×2mm. 3 , the embodiments of this application are not specifically limited.

[0055] The magnetic field sensing system 102 is a system for sensing the trajectory of a moving object using magnetic field signals. In the embodiment of the present application, the magnetic field sensing system 102 includes n×n magnetic induction sensors, where n is an integer greater than 1. The n×n magnetic induction sensors form an n×n square array.

[0056] For example, Figure 3 The above is a schematic diagram of the deployment of the magnetic field sensing system 102. The magnetic field sensing system 102 includes 9 magnetic induction sensors, namely magnetic induction sensors 1 to 9, forming a 3×3 square array, covering the operating range corresponding to the acupuncture process of the needle.

[0057] It should be noted that the sampling frequency of the magnetic induction sensor can be set as needed, for example, the sampling rate is 80 Hz.

[0058] Thus, the magnetic field sensing system 102 can be combined with the micro-permanent magnet 101-2 in the needle 101 to acquire magnetic data of the moving object during its motion. It is understood that the magnetic data collected by the acquisition device comprising the magnetic field sensing system 102 and the micro-permanent magnet provided in this application is not easily obstructed and is not affected by changes in lighting or image blur. Therefore, utilizing this magnetic data can supplement the deficiencies in video frame data and greatly improve the accuracy of locating the motion trajectory of the moving object. Therefore, it is particularly suitable for small, fast, and frequently obstructed motion scenarios such as acupuncture.

[0059] It should be noted that the plane where the magnetic induction sensor is located is the XY plane, see the attached Figure 3 shown.

[0060] Next, the magnetic field sensing system 102 is used to send the collected magnetic data to the processor 104 .

[0061] The video capture device 103 is used to capture video frame data, including that of a moving object during its motion. Considering that a single video capture device 103 may have issues such as occlusion and coverage, and inaccurate depth information, resulting in poor accuracy in the motion trajectory obtained from the captured video frame data, in this embodiment of the application, the video capture device 103 may include multiple video capture devices to capture the moving object from different shooting angles.

[0062] For example, the Figure 4 Schematic diagram of a dual-view video capture device provided in an embodiment of the present application. The dual-view video capture device includes a first camera 401 and a second camera 402. The first camera 401 and the second camera 402 capture the acupuncture process of the needle 101 from different shooting angles.

[0063] The first camera 401 and the second camera 402 provided in the embodiment of the present application are both depth cameras.

[0064] It should be noted that the resolutions and frame rates of the two cameras provided in the embodiments of the present application can be the same or different. For example, both cameras meet the following requirements: RGB frame resolution of 1920×1080, frame rate of 30fps, depth output resolution of 1280×720, frame rate of 90fps.

[0065] In a specific implementation, the first camera 401 and the second camera 402 are installed at positions that ensure that the shooting angles of the two cameras cover the entire motion area and avoid blind spots. Figure 4 As shown, the included angle α of the shooting angles of the first camera 401 and the second camera 402 may be 60° to 90°.

[0066] During actual filming, to improve the accuracy of determining the spatial observation trajectory from the video frame data, the first camera 401 and the second camera 402 need to be triggered synchronously. That is, the trajectory capture device 400 may include a synchronization module that synchronously triggers the first camera and the second camera to capture video frame data corresponding to the acupuncture process.

[0067] Furthermore, to ensure calculation accuracy, the camera can be calibrated to obtain the internal and external parameters of the camera.

[0068] Among them, the internal and external parameters of the camera include the intrinsic parameters of the camera and the external parameters of the camera. The intrinsic parameters of the camera include the focal length, principal point position and distortion parameters of the camera. In the embodiment of the present application, the intrinsic parameters of the camera are obtained by shooting a series of calibration plates of known sizes (such as a chessboard). The external parameters of the camera refer to the relative position and posture between the first camera 401 and the second camera 402. Among them, calibration can be performed by measuring the rotation matrix and transformation matrix between the two cameras.

[0069] The camera projection model is shown in formula (1).

[0070]

[0071] Where: [R|T] is the rotation matrix and translation vector from the world coordinate system to the camera coordinate system; K is the camera's intrinsic parameter matrix; [X, Y, Z] T is the coordinate of the object in the world coordinate system; [x′, y′] T is the coordinate of the object in the image coordinate system, and z' is the depth of the object from the camera plane.

[0072] It can be understood that through camera calibration, the actual world coordinates corresponding to each pixel can be calculated, and then the three-dimensional position of the object can be obtained.

[0073] The video acquisition device 103 sends the acquired video frame data to the processor 104. Correspondingly, the processor 104 acquires the video frame data, and then acquires the motion trajectory of the moving object during the motion process based on the magnetic data and the video frame data.

[0074] Next, the professional terms involved in the embodiments of this application are explained.

[0075] (1) Magnetic data

[0076] Magnetic data refers to magnetic data obtained through magnetic positioning technology and is used to determine the spatial predicted trajectory of a moving object. In one example, the magnetic data can be the magnetic field strength obtained by a magnetic induction sensor.

[0077] For example, assuming that the micro permanent magnet is cylindrical, the micro permanent magnet will be modeled as a dipole model. Figure 5 This is a schematic diagram of a dipole model provided in an embodiment of the present application, where the position of the micro permanent magnet is (a, b, c) and the posture is (m, n, p).

[0078] Continue to see Figure 5 , the position of the i-th magnetic induction sensor is expressed as (x i ,y i ,z i ), H0 is a unit vector (m,n,p) T , represents the direction of the micro permanent magnet from the south pole to the north pole. The distance vector between the micro permanent magnet and the i-th magnetic induction sensor is F i =(x i -a,y i -b,z i -c). (x i ,y i ,z i ) magnetic field strength B i It can be expressed as formula (2):

[0079]

[0080] Where N is the total number of sensors, B T is a constant related to the micro permanent magnet and can be expressed as formula (3):

[0081]

[0082] Among them, μ0 is the magnetic permeability of air, μ r is the relative magnetic permeability of the micro permanent magnet, δ and L represent the radius and length of the micro permanent magnet respectively; M0 is the magnetization intensity on the surface of the micro permanent magnet.

[0083] Among them, R i is the distance from the micro permanent magnet to the i-th magnetic induction sensor, as shown in formula (4):

[0084]

[0085] In an embodiment of the present application, magnetic data is used to obtain the motion trajectory of a moving object during its motion.

[0086] For example, see Figure 5 As shown, the magnetic field strength B of N magnetic induction sensors can be i ' to determine the position (a, b, c) and posture (m, n, p) of the micro permanent magnet. It can be understood that the position and posture of the micro permanent magnet are the same as the position and posture of the acupuncture needle end.

[0087] In a specific implementation, the position and attitude of the micro permanent magnet can be obtained based on a nonlinear optimization method. Specifically, the objective function is to minimize the error between the measured value and the theoretical value, as shown in formula (5):

[0088]

[0089] Here, B i is the theoretical magnetic field strength value (calculated by the magnetic dipole model). i ′ is the actual magnetic field strength value measured by the magnetic induction sensor. (x i ,y i ,z i ) magnetic field strength B i =(B ix ,,B iy ,,B iz ), triaxial component B ix ,B iy ,B iz It can be calculated by formula (6):

[0090]

[0091] It should be noted that in the embodiments of the present application, the Levenberg-Marquardt (LM) algorithm can be used for iterative optimization, wherein the initial values ​​can be obtained based on the particle swarm optimization algorithm. During the iteration process, the position of the previous frame is used as the initial value, and the position and direction of the current frame are iteratively solved. Other methods for iterative optimization can also be used, and are not specifically limited in the embodiments of the present application.

[0092] (2) Video frame data

[0093] Video frame data refers to data consisting of a series of video frames.

[0094] In the embodiment of the present application, the video frame data is acquired by a video capture device and includes multiple video frames of a moving object during its motion. For example, the video frame data refers to a video of the operation of a needle during acupuncture captured by the video capture device.

[0095] In an embodiment of the present application, the video frame data can be used to determine the spatial observation trajectory of a moving object. Specifically, the feature points of the moving object in the video frame data can be extracted based on an image processing algorithm, and the spatial observation trajectory of the moving object can be obtained by obtaining the spatial position of the moving object at each acquisition moment.

[0096] For example, continue Figure 4 As shown, the spatial position of the needle during acupuncture can be obtained through the stereoscopic video or image information captured by the first camera 401 and the second camera 402. The video frames captured at the same time are dual-view video frames, that is, they include the first video frame captured by the first camera 401 and the second video frame captured by the second camera 402.

[0097] It should be noted that first camera 401 and second camera 402 are parallel. That is, in this embodiment of the present application, dual-view video frames are acquired using first camera 401 and second camera 402. These dual-view video frames use parallax to calculate the position of an object in three-dimensional space (i.e., spatial observation position). Parallax refers to the difference in the horizontal coordinates of a feature point in the first video frame and a feature point in the second video frame at the same spatial location. A detailed analysis is provided below.

[0098] For example, Figure 6 This is a schematic diagram of obtaining the spatial observation position of a needle provided in an embodiment of the present application. Specifically, the following steps are included:

[0099] Step 1: Camera Settings.

[0100] In the embodiment of the present application, the camera setting includes two parts: parallax calculation and camera calibration. Among them, camera calibration includes calibrating the intrinsic and extrinsic parameters of the camera. For details, please refer to the description of camera calibration mentioned above and will not be repeated here.

[0101] Disparity calculation refers to calculating disparity.

[0102] Continue to see Figure 4 , for the first moment, the feature point p of the first video frame captured by the first camera 401 L (x L ,y L ) and the feature point p of the second video frame captured by the second camera R (x R ,y R ), where the feature point p L (x L ,y L ) and feature point p R (x R ,y R ), corresponding to the same spatial position point, the definition of its parallax d is shown in formula (7):

[0103] d=x L -x R (7)

[0104] Based on the parallax, the camera geometry model (also known as stereo geometry) is used to calculate the depth information of the object. The depth information refers to the distance between the object and the camera. The depth calculation formula is shown in formula (8):

[0105]

[0106] Where Z is the depth from the object point to the camera, f is the focal length of the camera, and B is the baseline length between the two cameras (i.e., the distance between the two cameras).

[0107] Step 2: Image processing and analysis.

[0108] The video frame data can be processed through feature point extraction and feature matching.

[0109] In the embodiment of the present application, the feature point extraction indicates extracting feature points of a needle in the video frame data.

[0110] During acupuncture, the needle is usually composed of multiple feature points, such as the edge of the needle handle, the top of the needle body, etc., which can be extracted through edge detection, corner detection and / or scale-invariant feature transformation.

[0111] For example, in edge detection, the Canny edge detection operator can be used to find the outline of an object by detecting the areas with the most dramatic grayscale changes in the image. In corner detection, the Harris corner detection algorithm can be used to find areas with large angle changes in the image, which are usually at the corners of an object or at prominent surface features.

[0112] Feature point matching involves matching feature points at the same spatial location in two video frames at the same moment. For example, at the first moment, the first camera captures the first video frame, and the second camera captures the second video frame. After extracting feature points from each video frame, the feature points of the first video frame can be matched with the feature points of the second video frame to obtain at least one matching pair. The two feature points in a matching pair are located at the same spatial location.

[0113] In an embodiment of the present application, the processor may establish a correspondence between two video frames through stereo matching, wherein the stereo matching includes but is not limited to feature point or epipolar constraints corresponding to scale-invariant feature transformation.

[0114] Step 3: Calculate the three-dimensional coordinates of the needle.

[0115] The three-dimensional coordinates of the needle refer to the spatial position of the needle at a given moment.

[0116] After obtaining the disparity and feature matching, the three-dimensional coordinates of the object (needle) can be calculated by triangulation. The matching feature points p have been extracted from the video frames of the first camera and the second camera. L , p R , the corresponding spatial position point P(X, Y, Z), i.e. the three-dimensional coordinates of the needle, can be calculated by the following formula (9):

[0117]

[0118] Where: (x L ,y L ) is the coordinate of the matching point in the first camera; (c x ,c y ) is the coordinate of the principal point of the first camera; f is the focal length of the first camera; B is the baseline length of the first camera; d is the parallax.

[0119] The data processing method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0120] Attachment Figure 7 A data processing method flow chart provided in an embodiment of the present application is applied to the attached Figure 1 The processor, the method comprises the following:

[0121] S710: The processor obtains magnetic data and video frame data during the motion process.

[0122] The magnetic data is acquired through magnetic positioning technology and is used to determine the predicted spatial trajectory of the moving object during its motion. For example, the magnetic field sensing system sends the acquired magnetic data to the processor, which enables the processor to acquire the magnetic data during the motion process.

[0123] The video frame data is acquired by a video capture device and includes multiple video frames of a moving object during motion. For example, a first camera and a second camera, both of which have calibrated camera intrinsic parameters and camera extrinsic parameters, acquire dual-view video frame data and send the video frame data to the processor.

[0124] S720: The processor obtains a motion trajectory of the moving object during the motion process according to the magnetic data and the video frame data.

[0125] The processor may fuse the magnetic data and the video frame data and, based on the fused data, obtain a motion trajectory. Specifically, the processor first obtains a predicted spatial trajectory of the moving object during motion based on the magnetic data, and obtains an observed spatial trajectory of the moving object during motion based on the video frame data. The processor may then fuse the predicted spatial trajectory and the observed spatial trajectory, using the fused trajectory as the motion trajectory of the moving object.

[0126] For example, if the predicted spatial trajectory of the moving object is Tpre and the observed spatial trajectory of the moving object is T wat , the fused motion trajectory T true =a*Tpre+b*T wat , where a and b are both integers greater than 0 and less than 1, and a+b=1.

[0127] To further improve the fusion accuracy, the processor can fuse the spatial prediction trajectory and the spatial observation trajectory based on the Kalman filter algorithm. Specifically, the fusion weights a and b are dynamically obtained based on the Kalman filter algorithm. Among them, the fusion weight b is the Kalman gain KK k .

[0128] It should be noted that in order to ensure the acquisition accuracy, first, the transformation relationship TT between the sensor coordinate system and the camera coordinate system is established m->v , allowing both measurement results to be unified into the same coordinate system. Once established, the needle position measured by the sensor can be directly converted to the camera coordinate system and aligned with the video frame data. Next, to ensure that the collected magnetic data and video frame data are at the same instant, that is, to ensure time synchronization, hardware triggering or software timestamps can be used to ensure synchronization between the start of magnetic data acquisition and the start of video frame data.

[0129] The following takes the magnetic data and video frame at the kth moment as an example to obtain the Kalman gain KK in detail k Where k is a positive integer.

[0130] First, the processor defines a state vector. In the embodiment of the present application, the state vector includes the spatial position and posture of the needle at the first moment. For example, the state vector XX k It is defined as formula (10):

[0131] x k =[x k y k z k θ x θ y θ z ] T (10)

[0132] Where: x k ,y k ,z k is the position coordinate of the needle body in three-dimensional space at the first moment; θ x ,θ y ,θ z is the posture of the needle, expressed as the rotation angle of the needle in Euler angles.

[0133] The processor then makes a state vector prediction based on the magnetic data.

[0134] Considering that the position of the needle body changes little in a short time, the processor uses the state transition model to obtain the state vector at the current moment (i.e., the kth moment) based on the state vector obtained at the previous moment (k-1th moment). The state transition model is based on the assumption that the movement of the needle is stable, that is, the position does not change in a short time, and uses the unit matrix as the state transition matrix F. F is shown in formula (11):

[0135]

[0136] Based on the state vector x at the last moment k-1 , predict the current state vector x k|k-1 , the specific prediction formula is shown in formula (12):

[0137] x k|k-1 =F·x k-1 (12)

[0138] The processor uses the state covariance matrix to describe the uncertainty of the state vector prediction, where the state covariance matrix P k|k-1 Updated by the following formula (13):

[0139] P k|k-1=FP k-1 F T +Q (13)

[0140] Where Q is the process noise covariance matrix, which represents the error of the system model, and P K-1 is the state covariance matrix at the previous moment.

[0141] Finally, the processor uses the video frame data to correct the predicted value. k ===[xx vis,，k yy vis,,k zz vis,,k ] T , the update process is as follows:

[0142] The processor determines the spatial observation position corresponding to the state vector based on the observation model. Specifically, for binocular vision video frames, the observation model refers to a model in which the observation matrix H is the unit matrix. The observation matrix H is specifically shown in formula (14):

[0143]

[0144] The processor calculates the Kalman gain K k , the specific calculation formula is shown in formula (15):

[0145] K k =P k|k-1 ·H T ·(H·P k|k-1 ·H T +R) -1 (15)

[0146] Where R is the covariance matrix of the observation noise, which represents the noise of the video frame data.

[0147] To increase the Kalman gain K k The processor uses the Kalman gain to correct the predicted value and obtains the estimated value as shown in formula (16):

[0148] x k =x k|k-1 +K k ·(z k -H·x k|k-1 ) (16)

[0149] Finally, the covariance matrix is ​​updated based on formula (17), which is as follows:

[0150] P k =(IK k ·H)·P k|k-1 (17)

[0151] Among them, the Kalman gain K k The larger the value is, the greater the weight of the video frame data is. Therefore, through the Kalman gain, the fusion weight of the spatial observation trajectory corresponding to the video frame data and the spatial prediction trajectory corresponding to the magnetic data is dynamically adjusted, thereby obtaining a high-precision motion trajectory of the needle during the acupuncture process.

[0152] For example, Figure 8 A schematic diagram of a motion trajectory acquisition method provided in an embodiment of the present application. The Kalman filter algorithm is used to fuse the magnetic positioning prediction trajectory and the visual observation trajectory. The Kalman filter trajectory obtained has a small error compared to the actual observation trajectory. The magnetic positioning prediction trajectory is a spatial prediction trajectory, and the visual observation trajectory is a spatial observation trajectory.

[0153] Furthermore, Figure 9 This is a specific implementation diagram of another data processing method provided in an embodiment of the present application, wherein a sensor collects magnetic field and a binocular camera collects video frame data. The method includes the following contents:

[0154] S910: The processor starts a collection operation.

[0155] In an embodiment of the present application, the acquisition operation can be started by the processor, and the processor sends an acquisition instruction to the magnetic induction sensor and the binocular camera, and the magnetic induction sensor and the binocular camera start acquisition.

[0156] S920: The magnetic induction sensor collects magnetic data and sends the collected data to the processor.

[0157] S930: The binocular camera collects video frame data and sends the collected data to the processor.

[0158] S940: Processor data synchronization and time alignment.

[0159] Data synchronization involves aligning the sensor and camera coordinate systems and converting them to the same coordinate system for processing. Time alignment involves matching the timestamps of the sensor and video acquisition system to ensure that magnetic data and video frame data are captured at the same moment.

[0160] S950: Based on the Kalman filter algorithm, the magnetic data and video frame data are integrated to obtain the motion trajectory of the needle during the acupuncture operation.

[0161] S960: Output the trajectory and / or save the motion trajectory.

[0162] The present application provides a data processing method for acquiring magnetic data and video frame data, and then acquiring the motion trajectory of the moving object based on the magnetic data and video frame data. That is to say, the present application acquires the motion trajectory of the moving object through the magnetic data acquired by the magnetic positioning technology and the video frame data acquired by the video acquisition device. The present application does not need to rely on the experience and judgment of the observer, and is not easily interfered with by external factors. Moreover, the present application can make full use of the high-frequency sampled magnetic data of the magnetic positioning technology to dynamically predict and compensate for the low-frame-rate video frame data. Compared with acquiring the spatial position of the moving object using a single video frame data, a more accurate motion trajectory can be acquired, thus helping to improve the positioning accuracy of the motion trajectory of the moving object during the motion process.

[0163] In addition, an embodiment of the present application also provides a data processing device.

[0164] Attachment Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The device 1000 includes:

[0165] An acquisition unit 1001 is configured to acquire magnetic data and video frame data during a motion process; the magnetic data is acquired using magnetic positioning technology and is used to determine a predicted spatial trajectory of a moving object during the motion process; and the video frame data is acquired by a video acquisition device and includes video frames of the moving object.

[0166] The determining unit 1002 is configured to obtain a motion trajectory of the moving object during the motion process according to the magnetic data and the video frame data.

[0167] Optionally, the acquisition unit 1001 includes m video acquisition devices, and the acquisition unit 1001 is specifically configured to:

[0168] The video frame data is acquired through the m video acquisition devices, where m is an integer greater than 2, and each of the m video acquisition devices has a different shooting angle.

[0169] Optionally, if the m video acquisition devices include a first camera and a second camera, the apparatus 1000 further includes a calibration unit, which is configured to: before acquiring the first video frame data through the m video acquisition devices, calibrate the first camera and the second camera respectively, obtain the internal parameters of the first camera and the internal parameters of the second camera, and obtain the relative position and posture between the first camera and the second camera.

[0170] Optionally, the determining unit 1002 is specifically configured to:

[0171] determining a predicted spatial trajectory of the moving object during the movement based on the magnetic data; and determining an observed spatial trajectory of the moving object during the movement based on the video frame data;

[0172] The spatial prediction trajectory and the spatial observation trajectory are fused to obtain the motion trajectory of the moving object during the motion process.

[0173] Optionally, if the video frame data includes video frames at multiple moments, determining the spatial observation trajectory of the moving object during the movement based on the video frame data includes:

[0174] For each video frame at the multiple moments, execute:

[0175] Extracting feature points of a video frame; the video frame includes a first sub-video frame acquired by the first camera and a second sub-video frame acquired by the second camera;

[0176] Matching feature points of the first sub-video frame with feature points of the second sub-video frame to obtain at least one matching pair, wherein each matching pair in the at least one matching pair corresponds to the same spatial position point;

[0177] Obtaining a disparity corresponding to each matching pair in the at least one matching pair; obtaining a spatial observation position of the moving object based on the disparity corresponding to each matching pair, wherein the disparity corresponding to a first matching pair indicates a difference between horizontal coordinates of two feature points of the first matching pair;

[0178] The spatial observation trajectory is obtained based on the spatial observation position of the moving object at each moment.

[0179] Optionally, fusing the predicted spatial trajectory and the observed spatial trajectory to obtain the motion trajectory of the moving object during the motion process includes:

[0180] The spatial prediction trajectory and the spatial observation trajectory are fused through the Kalman filter algorithm to obtain the motion trajectory of the moving object in the motion process.

[0181] Optionally, if the moving object is acupuncture, the magnetic data is the magnetic field strength of the magnetic induction sensor during the movement, and obtaining the magnetic data includes:

[0182] The first magnetic data is obtained based on the micro permanent magnet at the acupuncture end and the magnetic field sensing system. The magnetic field sensing system includes n×n magnetic induction sensors, where n is an integer greater than 1. The n×n magnetic induction sensors are an n×n square array and are located in the moving area of ​​the moving object.

[0183] The present application provides a data processing device that obtains the motion trajectory of a moving object through magnetic data obtained by magnetic positioning technology and video frame data obtained by video acquisition equipment. The present application does not need to rely on the experience and judgment of the observer, and is not easily interfered with by external factors. In addition, the present application can make full use of the high-frequency sampled magnetic data of the magnetic positioning technology to dynamically predict and compensate for the low-frame-rate video frame data. Compared with using a single video frame data to obtain the spatial position of the moving object, a more accurate motion trajectory can be obtained, thereby helping to improve the positioning accuracy of the motion trajectory of the moving object during the movement process.

[0184] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, which, when running on a computer, enables the computer to execute the various steps or processes executed by the network device and terminal device in any of the aforementioned method embodiments.

[0185] According to the method provided in the embodiments of the present application, the present application also provides a computer-readable storage medium, which stores program code. When the program code runs on a computer, the computer executes the various steps or processes performed by the network device and terminal device in any of the aforementioned method embodiments.

[0186] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0187] In the embodiments of this application, each term and English abbreviation is provided for convenience of description and shall not constitute any limitation to this application. This application does not exclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0188] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part.

[0189] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0190] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0191] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire magnetic data and video frame data of the motion process; the magnetic data is acquired by magnetic positioning technology and is used to determine the spatial prediction trajectory of the moving object during the motion process; the video frame data is acquired by a video acquisition device and includes video frames of the moving object; The motion trajectory of the moving object during the motion process is acquired according to the magnetic data and the video frame data.

2. The acquisition method according to claim 1, characterized in that The method for obtaining the video frame data includes: The video frame data is acquired through m video acquisition devices, where m is an integer greater than 2, and each of the m video acquisition devices has a different shooting angle.

3. The acquisition method according to claim 2, characterized in that If the m video acquisition devices include a first camera and a second camera, before acquiring the first video frame data through the m video acquisition devices, the method further includes: The first camera and the second camera are calibrated respectively to obtain intrinsic parameters of the first camera and the second camera, and to obtain a relative position and posture between the first camera and the second camera.

4. The acquisition method according to claim 1, characterized in that The method further comprises: determining a predicted spatial trajectory of the moving object during the movement based on the magnetic data; and determining an observed spatial trajectory of the moving object during the movement based on the video frame data; The acquiring, according to the magnetic data and the video frame data, a motion trajectory of the moving object during the motion process, includes: The spatial prediction trajectory and the spatial observation trajectory are fused to obtain the motion trajectory of the moving object during the motion process.

5. The acquisition method according to claim 4, characterized in that: If the video frame data includes video frames at multiple moments, determining the spatial observation trajectory of the moving object during the movement based on the video frame data includes: For each video frame at the multiple moments, execute: Extracting feature points of a video frame; the video frame includes a first sub-video frame acquired by the first camera and a second sub-video frame acquired by the second camera; Matching feature points of the first sub-video frame with feature points of the second sub-video frame to obtain at least one matching pair, wherein each matching pair in the at least one matching pair corresponds to the same spatial position point; Obtaining a disparity corresponding to each matching pair in the at least one matching pair; obtaining a spatial observation position of the moving object based on the disparity corresponding to each matching pair, wherein the disparity corresponding to a first matching pair indicates a difference between horizontal coordinates of two feature points of the first matching pair; The spatial observation trajectory is obtained based on the spatial observation position of the moving object at each moment.

6. The acquisition method according to claim 4, characterized in that: The fusing of the spatial prediction trajectory and the spatial observation trajectory to obtain the motion trajectory of the moving object during the motion process includes: The spatial prediction trajectory and the spatial observation trajectory are fused through the Kalman filter algorithm to obtain the motion trajectory of the moving object in the motion process.

7. The acquisition method according to claim 1, characterized in that If the moving object is acupuncture, the magnetic data is the magnetic field strength of the magnetic induction sensor during the movement, and obtaining the magnetic data includes: The first magnetic data is obtained based on the micro permanent magnet at the acupuncture end and the magnetic field sensing system. The magnetic field sensing system includes n×n magnetic induction sensors, where n is an integer greater than 1. The n×n magnetic induction sensors are an n×n square array and are located in the moving area of ​​the moving object.

8. A data processing device, characterized in that: The device comprises: an acquisition unit, configured to acquire magnetic data and video frame data of a motion process; the magnetic data is acquired through magnetic positioning technology and is used to determine a predicted spatial trajectory of a moving object during the motion process; the video frame data is acquired by a video acquisition device and includes video frames of the moving object; A determination unit is configured to obtain a motion trajectory of the moving object during the motion process based on the magnetic data and the video frame data.

9. The processing device according to claim 8, characterized in that The acquisition unit includes m video acquisition devices, and the acquisition unit is specifically used to: The video frame data is acquired through the m video acquisition devices, where m is an integer greater than 2, and each of the m video acquisition devices has a different shooting angle.

10. A trajectory capture device, characterized in that: The device includes: a needle, a magnetic field sensing system, a video acquisition device and a processor; the magnetic field sensing system is connected to the processor, and the video acquisition device is connected to the processor; The needle tip includes a micro permanent magnet; The magnetic field sensing system includes n×n magnetic induction sensors, where n is an integer greater than 1, and the n×n magnetic induction sensors form an n×n square array; The magnetic field sensing system is used to obtain magnetic data of the movement process based on the micro permanent magnet; the magnetic data is used to determine the spatial prediction trajectory of the micro permanent magnet during the movement process; and the magnetic data is sent to the processor; The video acquisition device is used to obtain video frame data including the needle and send the video frame data to the processor; The processor is used to obtain the motion trajectory of the needle during the motion process according to the magnetic data and the video frame data.

Images