Remote Surgery Guidance Visualization Artificial Selection Site Tracking Method, System and Device

Abstract

The present invention provides a tracking method, system and device for visually manually selecting sites in remote surgical guidance, which relates to the technical field of remote surgical guidance. In the present invention, the target sites that need to be dynamically tracked are marked on the selected initial frame image, and then through real-time transmission, accurate output is achieved on the surgical end interaction interface, and through related technologies such as target tracking, real-time dynamic tracking of the target sites is achieved during the subsequent surgical process, effectively improving the surgical positioning accuracy, reducing surgical trauma, optimizing the surgical path and increasing the surgical success rate; improving the accuracy of remote surgical guidance and shortening the remote surgical time.

Description

Technical Field

[0001] The present invention relates to the technical field of remote surgical guidance, and particularly relates to a tracking method, system and device for visually manually selected sites in remote surgical guidance. Background Art

[0002] China has a large population, vast territory, unbalanced regional economic development, uneven distribution of medical resources, and limited medical levels of grass-roots medical personnel. The implementation of remote surgical guidance is of great social and economic benefits for solving the uneven distribution of medical resources, enabling patients in remote areas to enjoy high-level medical resources, and appropriately resolving the contradictions between the medical service demand and supply in remote areas, the uneven allocation of medical resources, the relative scarcity of resources in grass-roots medical institutions, and the structural differences in the educational level and service capabilities of health personnel.

[0003] Some grass-roots medical personnel have limited medical levels, and when facing surgeries with higher difficulties, the surgical quality cannot be fully guaranteed. Moreover, doctors with different qualifications also have different judgment criteria for the planning and selection of surgical paths. Remote surgical guidance can help the surgeons at the surgical end to perform surgeries smoothly and accurately under the guidance of the remote guiding doctors through interactive means, ensuring the safety, accuracy and quality of the surgeries. For example, based on a remote conferencing system, the remote doctor annotates and guides the surgery according to the surgical scene images captured by the on-site camera, and the remote conferencing system transmits the guiding annotations to the monitor in the operating room and conducts intraoperative guidance in cooperation with the voice.

[0004] However, the existing remote surgical guidance does not consider that in complex in-vivo environments such as the flexible deformation, occlusion and disappearance, scale change, light change and image blurring of human tissues, the surgical images change in real time and dynamically, and the accuracy, safety and quality of the subsequent surgical process at the surgical end cannot be effectively guaranteed. Summary of the Invention

[0005] (1) Technical Problems to be Solved

[0006] Aiming at the deficiencies of the prior art, the present invention provides a tracking method, system and device for visually manually selected sites in remote surgical guidance, and solves the technical problem that the real-time dynamic tracking of the surgical positions determined by the guiding doctors at the guiding end cannot be achieved.

[0007] (2) Technical Solutions

[0008] To achieve the above object, the present invention is realized through the following technical solutions:

[0009] A tracking method for visually manually selected sites in remote surgical guidance, comprising:

[0010] S1. Obtain the source point to be dynamically tracked as the target site according to the manual selection operation of the guiding doctor on the initial frame image, where the initial frame image is acquired from the shared screen of the remote guiding end and the surgical end;

[0011] S2. Obtain the target area containing the target site, perform a masking operation on the target area and detect sift feature points to obtain the first feature point set;

[0012] S3. Track the target area in real time, obtain the tracking area corresponding to each subsequent frame image of the initial frame image, perform a masking operation on the tracking area and detect sift feature points to obtain the second feature point set;

[0013] S4. According to the first feature point set and the second feature point set, obtain the coordinate position of the final matching point by performing feature matching and noise point elimination, and determine the homography matrix;

[0014] S5. Map the coordinate position of the target site using the homography matrix to determine the tracking site corresponding to each subsequent frame image of the initial frame image, and the tracking site is used to guide the surgical doctor to perform surgical operations.

[0015] Preferably, the tracking method further includes:

[0016] S6. Send the tracking site to the remote guiding end, and realize real-time communication between the guiding doctor and the surgical doctor through audio and video interaction.

[0017] Preferably, in S3, a fully convolutional Siamese network is used to track the target area in real time to obtain the tracking area corresponding to each subsequent frame image of the initial frame image, which specifically includes:

[0018] S31. Extract features from the target area of the initial frame image and each subsequent frame image of the initial frame image to obtain the corresponding feature maps;

[0019] S32. Calculate the similarity between the target area and the current frame image. Use the feature map of the target area as the convolutional kernel to perform a convolutional operation on the feature map of the current frame image to obtain the similarity score;

[0020] S33. When taking the maximum similarity score, take the area corresponding to the current frame image as the tracking area.

[0021] Preferably, S4 specifically includes:

[0022] S41. According to the first feature point set and the second feature point set, use the k-nearest neighbor fast matching algorithm to perform feature matching to obtain the initial matching points;

[0023] S42. Filter out noise points using the iForest algorithm based on the initial matching points to obtain the final matching points;

[0024] S43. Determine whether the number of the final matching points is greater than a set threshold. If so, obtain the homography matrix based on the coordinate positions of the final matching points.

[0025] Preferably, S43 further includes:

[0026] If the number of the final matching points is less than or equal to the set threshold, update the current frame image to the initial frame image and transfer to S32.

[0027] Preferably, in S2, obtaining the target region including the target site specifically includes:

[0028] Select a region with clear tissue texture including the target site on the initial frame image as the target region.

[0029] A tracking system for visual artificial selection sites in remote surgical guidance includes:

[0030] A first acquisition module, configured to obtain a source point to be dynamically tracked as a target site according to the manual selection operation of the doctor at the guidance end on the initial frame image, where the initial frame image is acquired from the shared screen of the remote guidance end and the surgical end;

[0031] A second acquisition module, configured to obtain a target region including the target site, perform a masking operation on the target region, and detect sift feature points to obtain a first feature point set;

[0032] Real-time track the target region, obtain the tracking region corresponding to each subsequent frame image of the initial frame image, perform a masking operation on the tracking region, and detect sift feature points to obtain a second feature point set;

[0033] A tracking module, configured to obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination according to the first feature point set and the second feature point set, and determine the homography matrix;

[0034] A mapping module, configured to map the coordinate position of the target site using the homography matrix to determine the tracking site corresponding to each subsequent frame image of the initial frame image, where the tracking site is used to guide the doctor at the surgical end to perform surgical operations.

[0035] Preferably, the tracking system further includes:

[0036] An interaction module, configured to send the tracking site to the remote guidance end and realize real-time communication between the doctor at the guidance end and the doctor at the surgical end through audio and video interaction.

[0037] A tracking device for visually manually selecting sites in remote surgical guidance, comprising a remote guidance end and a surgical end. The remote guidance end includes a first interaction device, and the surgical end includes a second interaction device and a sampling device;

[0038] The first interaction device is configured to receive the real-time surgical video collected by the sampling device as a shared screen between the remote guidance end and the surgical end, and obtain a source point to be dynamically tracked as a target site according to the manual selection operation of the doctor at the guidance end on the initial frame image, where the initial frame image is collected from the shared screen;

[0039] The second interaction device is configured to obtain a target area containing the target site, perform a masking operation and detect SIFT feature points on the target area, and obtain a first set of feature points;

[0040] And, it is configured to continuously track the target area, obtain a tracking area corresponding to each subsequent frame image of the initial frame image, perform a masking operation and detect SIFT feature points on the tracking area, and obtain a second set of feature points;

[0041] And, it is configured to, according to the first set of feature points and the second set of feature points, obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination, and determine a homography matrix

[0042] And, use the homography matrix to map the coordinate positions of the target site, determine a tracking curve corresponding to each subsequent frame image of the initial frame image, and the tracking curve is used to guide the doctor at the surgical end to perform surgical operations.

[0043] Preferably, the second interaction device is further configured to send the tracked site to the first interaction device;

[0044] Both the first interaction device and the second interaction device include an audio-video interaction device; the audio-video interaction device is used to enable real-time communication between the doctor at the guidance end and the doctor at the surgical end.

[0045] (III) Beneficial effects

[0046] The present invention provides a tracking method, system and device for visually manually selecting sites in remote surgical guidance. Compared with the prior art, it has the following beneficial effects:

[0047] The present invention marks the target sites that need dynamic tracking on the selected initial frame image, and then through real-time transmission, accurately outputs them on the surgical end interaction interface, and through related technologies such as target tracking, realizes the real-time dynamic tracking of the target sites during the subsequent surgical process, effectively improving the surgical positioning accuracy, reducing surgical trauma, optimizing the surgical path and increasing the surgical success rate; improving the accuracy of remote surgical guidance and reducing the remote surgical time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0049] Figure 1 It is a schematic flowchart of a method for tracking visually selected sites in remote surgical guidance provided by an embodiment of the present invention;

[0050] Figure 2 It is a structural block diagram of a system for tracking visually selected sites in remote surgical guidance provided by an embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of the overall business process of a device for tracking visually selected sites in remote surgical guidance provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0053] By providing a method, system, and device for tracking visually selected sites in remote surgical guidance, the embodiments of the present application solve the technical problem of being unable to achieve real-time dynamic tracking of the surgical positions determined by the doctors at the guidance end, and effectively improve the surgical positioning accuracy, reduce surgical trauma, optimize the surgical path, and increase the surgical success rate.

[0054] The general idea of the technical solutions in the embodiments of the present application to solve the above technical problems is as follows:

[0055] In the embodiment of the present invention, the target site to be dynamically tracked is marked on the selected initial frame image, and then through real-time transmission, it is accurately output on the surgical end interaction interface, and through related technologies such as target tracking, the real-time dynamic tracking of the target site in the subsequent surgical process is realized, effectively improving the surgical positioning accuracy, reducing surgical trauma, optimizing the surgical path and increasing the surgical success rate; improving the accuracy of remote surgical guidance and reducing the remote surgical time.

[0056] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the specification drawings and specific embodiments.

[0057] Embodiment:

[0058] In the first aspect, as Figure 1 shown, the embodiment of the present invention provides a method for tracking a visually manually selected site for remote surgical guidance, including:

[0059] S1. According to the manual selection operation of the doctor at the guidance end on the initial frame image, obtain the source point to be dynamically tracked as the target site, and the initial frame image is collected from the shared screen of the remote guidance end and the surgical end;

[0060] S2. Obtain the target area containing the target site, and perform a masking operation and detect sift feature points on the target area to obtain a first set of feature points;

[0061] S3. Real-time track the target area, obtain the tracking area corresponding to each subsequent frame image of the initial frame image, and perform a masking operation and detect sift feature points on the tracking area to obtain a second set of feature points;

[0062] S4. According to the first set of feature points and the second set of feature points, obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination, and determine the homography matrix;

[0063] S5. Use the homography matrix to map the coordinate position of the target site, and determine the tracking site corresponding to each subsequent frame image of the initial frame image, and the tracking site is used to guide the doctor at the surgical end to perform surgical operations;

[0064] S6. Send the tracking site to the remote guidance end, and realize real-time communication between the doctor at the guidance end and the doctor at the surgical end through audio and video interaction.

[0065] In the embodiments of the present invention, the target sites that need to be dynamically tracked are marked on the selected initial frame image, and then through real-time transmission, they are accurately output on the surgical end interaction interface. Through related technologies such as target tracking, real-time dynamic tracking of the target sites during the subsequent surgical process is achieved, effectively improving the surgical positioning accuracy, reducing surgical injuries, optimizing the surgical path, and increasing the surgical success rate; improving the accuracy of remote surgical guidance and shortening the remote surgical time.

[0066] The following will introduce each step of the above technical solution in detail in combination with specific content:

[0067] S1. According to the manual selection operation of the doctor at the guidance end on the initial frame image, obtain the source points that need to be dynamically tracked as the target sites, and the initial frame image is collected from the shared screen of the remote guidance end and the surgical end.

[0068] The remote guidance end freezes the target frame image frame suitable for marking the guidance path in the real-time shared screen of the surgical initiation end as the initial frame image to ensure that both parties are concerned about the same frame image.

[0069] The doctor at the distal guidance end starts the selection operation of the surgical planning source points on the initial frame image: use input devices such as a mouse and a stylus to mark at key positions, and obtain the source points that need to be dynamically tracked as the target sites.

[0070] S2. Obtain the target region containing the target sites, and perform a masking operation and detect sift feature points on the target region to obtain the first feature point set src_kpoints.

[0071] The obtaining of the target region containing the target sites in S2 specifically includes:

[0072] Select a region with clear tissue texture including the target sites on the initial frame image as the target region.

[0073] S3. Real-time track the target region, obtain the tracking region corresponding to each subsequent frame image of the initial frame image, and perform a masking operation and detect sift feature points on the tracking region to obtain the second feature point set dst_kpoints.

[0074] The real-time tracking of the target region in S3 by using a fully convolutional Siamese network to obtain the tracking region corresponding to each subsequent frame image of the initial frame image specifically includes:

[0075] S31. Extract features from the target region z of the initial frame image and each subsequent frame image x of the initial frame image to obtain the corresponding feature maps and

[0076] S32. Calculate the similarity between the target region and the current frame image. Using the feature map of the target region as the convolution kernel, perform convolution operation on the feature map of the current frame image to obtain the similarity score.

[0077]

[0078] Among them, the function represents a convolutional neural network including a convolutional layer and a pooling layer, equivalent to a feature extractor, for feature extraction of images; the function f(x, z) represents the similarity score between z and x, and the larger the score value, the more similar.

[0079] S33. When taking the maximum similarity score, use the region corresponding to the current frame image as the tracking region.

[0080] S4. According to the first feature point set and the second feature point set, obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination, and determine the homography matrix; specifically including:

[0081] S41. According to the first feature point set src_kpoints and the second feature point set dst_kpoints, use the k-nearest neighbor fast matching algorithm to perform feature matching to obtain the initial matching points.

[0082] For k-nearest neighbor matching:

[0083]

[0084] Among them, x ik represents the i-th feature point in the first feature point set src_kpoints; y jk represents the j-th feature point in the second feature point set dst_kpoints; K represents the sample dimension, which represents the length of the feature vector corresponding to the feature points in the first feature point set and the second feature point set here; d ij represents the distance between the feature points x ik and y jk ; D represents the distance space composed of the distances between all feature points in the second feature point set dst_kpoints and the i-th feature point in the first feature point set src_kpoints.

[0085] Determine the minimum distance d in the space D ij′, and it is explained that the \(j'\)-th feature point in the second feature point set \(dst\_kpoints\) is successfully matched with the \(i\)-th feature point in the first feature point set \(src\_kpoints\); similarly, the k-nearest neighbor matching algorithm is also applied to other feature points in \(src\_kpoints\) to obtain preliminary matching points.

[0086] S42. According to the initial matching points, the iForest algorithm is used to filter out noise points to obtain the final matching points.

[0087] For the iForest algorithm (Isolation Forest algorithm):

[0088] ① According to the data set \(N\) containing \(n\) samples, specifically in the embodiment of the present invention, that is, the first feature point set \(src\_kpoints\) including \(n\) samples, calculate the average path length of the isolation tree \(iTree\):

[0089]

[0090] where \(H(i)\) is the harmonic number, and this estimated value takes \(\ln(i)+0.577\).

[0091] ② Sample anomaly score:

[0092]

[0093] where \(h(x)\) is the number of edges passed by the sample point \(x\) in the initial matching points from the root node to the leaf node of the isolation tree \(iTree\); \(E(h(x))\) is the expectation of the path of the sample \(x\) in a batch of isolation trees.

[0094] ③ Anomaly determination:

[0095] When \(E(h(x))\to c(n)\), \(s\to0.5\), the average path length of the sample \(x\) is close to the average length of the tree, and the anomaly cannot be determined;

[0096] When \(E(h(x))\to0\), \(s\to1\), that is, the anomaly score of \(x\) is close to 1, and it is determined as an anomaly point;

[0097] When \(E(h(x))\to n - 1\), \(s\to0\), that is, the anomaly score of \(x\) is close to 0, and it is determined as a normal point.

[0098] When \(s\in(0,5]\), accept the initial matching point as the final matching point; otherwise, do not accept.

[0099] S43. Determine whether the number of the final matching points is greater than the set threshold. If so, obtain the homography matrix according to the coordinate positions of the final matching points.

[0100] For the homography matrix \(H\):

[0101]

[0102] Obtained:

[0103]

[0104] Among them, x1 represents the row vector composed of the x-axis coordinates of the feature points in the first feature point set src_kpoints; y1 represents the row vector composed of the y-axis coordinates of the feature points in the first feature point set src_kpoints; e1 represents the row vector with elements all being 1 and having the same length as x1.

[0105] x2 represents the row vector composed of the x-axis coordinates of the feature points in the second feature point set dst_kpoints; y2 represents the row vector composed of the y-axis coordinates of the feature points in the second feature point set dst_kpoints; e2 represents the row vector with elements all being 1 and having the same length as x2.

[0106] Since ||H|| = 1, the degree of freedom of the matrix H is 8. Therefore, at least 4 final matching point pairs are provided to solve the matrix H.

[0107] It should be noted that in the embodiments of the present invention, it is preferably to use a large number of successfully matched feature point pairs to improve the robustness of the algorithm, rather than being limited to the number of matching point pairs just for solving the matrix H.

[0108] If the number of the final matching points is less than or equal to the set threshold, update the current frame image to the initial frame image and transfer to S32.

[0109] S5. Use the homography matrix to map the coordinate positions of the target site, determine the tracking sites corresponding to each subsequent frame image of the initial frame image, and the tracking curve is used to guide the surgeon at the surgical end to perform surgical operations.

[0110] S6. Send the tracking sites to the remote guidance end, and realize real-time communication between the doctor at the guidance end and the doctor at the surgical end through audio and video interaction.

[0111] Specifically, the embodiments of the present invention use special color markings to track the sites, and transmit the processed image to the remote guidance end in real time through video communication. The doctor at the guidance end continuously outputs medical knowledge according to the current diagnosis and treatment plan to guide the doctor at the surgical receiving end to perform surgical operations.

[0112] The audio and video interaction can be specifically implemented by using audio and video devices such as microphones, speakers, and displays.

[0113] The tracking method provided by the embodiments of the present invention is mainly applied to, but not limited to, the remote pathology guidance scenario; in complex in-vivo environments such as the flexible deformation, occlusion and disappearance, scale change, illumination change, and image blurring of human tissues, it can perform visual input on the target site manually selected at the guidance end through a remote interaction device, and perform visual and precise output of the target site on the remote interaction device at the surgical end; the surgical end can accurately track the target site manually selected, realize the surgical planning position and guidance in remote surgical guidance, and guide the surgeon to perform precise surgical operations; it can perform real-time sharing of videos and images at the guidance end and the surgical end, and realize real-time interaction during the remote surgical guidance process through audio input and output devices.

[0114] In addition, for the convenience of later surgical evaluation, quality control, and providing a basis for the patient's later treatment, the embodiments of the present invention can store images and videos at the guidance end and the surgical end respectively.

[0115] In the second aspect, as Figure 2 shown, a tracking system for visually manually selected sites in remote surgical guidance includes:

[0116] A first acquisition module, configured to obtain the source point to be dynamically tracked as the target site according to the manual selection operation of the doctor at the guidance end in the initial frame image, and the initial frame image is acquired from the shared screen of the remote guidance end and the surgical end;

[0117] A second acquisition module, configured to obtain the target area containing the target site, perform a masking operation on the target area and detect sift feature points, and obtain a first set of feature points;

[0118] The target area is tracked in real time, the tracking area corresponding to each subsequent frame image of the initial frame image is obtained, a masking operation is performed on the tracking area and sift feature points are detected, and a second set of feature points is obtained;

[0119] A tracking module, configured to obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination according to the first set of feature points and the second set of feature points, and determine the homography matrix;

[0120] A mapping module, configured to map the coordinate position of the target site by using the homography matrix, and determine the tracking site corresponding to each subsequent frame image of the initial frame image, and the tracking site is used to guide the doctor at the surgical end to perform surgical operations;

[0121] An interaction module, configured to send the tracking site to the remote guidance end, and realize real-time communication between the doctor at the guidance end and the doctor at the surgical end through audio and video interaction.

[0122] In a third aspect, a tracking device for visually manually selecting sites in remote surgical guidance includes a remote guidance end and a surgical end. The remote guidance end includes a first interaction device, and the surgical end includes a second interaction device and a sampling device;

[0123] The first interaction device, as Figure 3 shown, is configured to receive the real-time surgical video collected by the sampling device as a shared screen between the remote guidance end and the surgical end, and obtain a source point to be dynamically tracked as a target site according to the manual selection operation of the doctor at the guidance end on the initial frame image, where the initial frame image is collected from the shared screen.

[0124] The first interaction device is further configured to transmit the path curve to the surgical end after encoding. Specifically, for the path curve selected by the doctor at the guidance end, it is encapsulated into Json data and transmitted to the second interaction device at the surgical end through the TCP protocol.

[0125] The second interaction device is configured to obtain a target area containing the target site, perform a masking operation and detect SIFT feature points on the target area to obtain a first set of feature points;

[0126] and is configured to continuously track the target area, obtain a tracking area corresponding to each subsequent frame image of the initial frame image, perform a masking operation and detect SIFT feature points on the tracking area to obtain a second set of feature points;

[0127] and is configured to, according to the first set of feature points and the second set of feature points, obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination, and determine a homography matrix

[0128] and is configured to map the coordinate position of the target site using the homography matrix to determine a tracking curve corresponding to each subsequent frame image of the initial frame image, where the tracking curve is used to guide the doctor at the surgical end to perform surgical operations.

[0129] Preferably, the second interaction device is further configured to send the tracking curve to the first interaction device in real time through a network protocol;

[0130] Both the first interaction device and the second interaction device include audio-video interaction devices; the audio-video interaction devices are used to enable real-time communication between the doctor at the guidance end and the doctor at the surgical end.

[0131] It is understandable that a tracking system and device for visually selecting sites for remote surgical guidance provided by an embodiment of the present invention correspond to a method for tracking visually selected sites for remote surgical guidance provided by an embodiment of the present invention. For the explanations, examples, beneficial effects, etc. of the relevant content, reference can be made to the corresponding parts in the method for tracking visually selected sites for remote surgical guidance, which will not be elaborated here.

[0132] In summary, compared with the prior art, the following beneficial effects are achieved:

[0133] 1. In the embodiment of the present invention, the target sites to be dynamically tracked are marked on the selected initial frame image, and then through real-time transmission, accurate output is achieved on the surgical end interaction interface. Through related technologies such as target tracking, real-time dynamic tracking of the target sites during the subsequent surgical process is realized, effectively improving the surgical positioning accuracy, reducing surgical trauma, optimizing the surgical path, and increasing the surgical success rate; improving the accuracy of remote surgical guidance and shortening the remote surgical time.

[0134] 2. The tracking method provided by the embodiment of the present invention is mainly applied to, but not limited to, the remote pathology guidance scenario; in complex in-vivo environments such as flexible deformation, occlusion and disappearance, scale change, illumination change, and image blur of human tissues, visual input of the manually selected target sites can be performed at the guidance end through a remote interaction device, and visual precise output of the target sites can be performed on the remote interaction device at the surgical end; the surgical end can accurately track the manually selected target sites, realizing the surgical planning position and guidance in remote surgical guidance, and guiding the surgeon to perform precise surgical operations; real-time sharing of videos and images can be performed at the guidance end and the surgical end, and real-time interaction during the remote surgical guidance process can be achieved through audio input and output devices.

[0135] 3. For the convenience of later surgical evaluation, quality control, and providing a basis for the patient's later treatment, the embodiment of the present invention can store images and videos at the guidance end and the surgical end respectively.

[0136] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0137] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tracking method for visually manually selecting sites in remote surgery guidance, characterized in that, Including: S1. According to the manual selection operation of the guiding doctor on the initial frame image, obtain the source point to be dynamically tracked as the target site, and the initial frame image is collected from the shared screen of the remote guiding end and the surgical end; S2. Obtain the target area containing the target site, perform a masking operation and detect sift feature points on the target area, and obtain the first feature point set; S3. Real-time track the target area, obtain the tracking area corresponding to each subsequent frame image of the initial frame image, perform a masking operation and detect sift feature points on the tracking area, and obtain the second feature point set; S4. According to the first feature point set and the second feature point set, obtain the coordinate position of the final matching point by performing feature matching and noise point elimination, and determine the homography matrix; S5. Map the coordinate position of the target site using the homography matrix to determine the tracking site corresponding to each subsequent frame image of the initial frame image; In S3, a fully convolutional Siamese network is used to real-time track the target area and obtain the tracking area corresponding to each subsequent frame image of the initial frame image. Specifically, it includes: S31. Extract features from the target area of the initial frame image and each subsequent frame image of the initial frame image to obtain the corresponding feature maps; S32. Calculate the similarity between the target area and the current frame image. Use the feature map of the target area as the convolution kernel to perform a convolution operation on the feature map of the current frame image to obtain the similarity score; S33. When taking the maximum similarity score, use the area corresponding to the current frame image as the tracking area; S4 specifically includes: S41. According to the first feature point set and the second feature point set, use the k-nearest neighbor fast matching algorithm to perform feature matching to obtain the initial matching points; S42. According to the initial matching points, use the iForest algorithm to filter noise points to obtain the final matching points; S43. Judge whether the number of the final matching points is greater than the set threshold. If so, obtain the homography matrix according to the coordinate positions of the final matching points; S43 further includes: If the number of the final matching points is less than or equal to the set threshold, update the current frame image to the initial frame image and transfer to S32; In S2, obtaining the target area containing the target site specifically includes: Select an area with clear tissue texture including the target site on the initial frame image as the target area.

2. The tracking method for visually manually selecting a site in remote surgery guidance according to claim 1, wherein, The tracking method further includes: S6. Send the tracking site to the remote guiding end, and realize real-time communication between the guiding doctor and the surgical doctor through audio and video interaction.

3. A tracking system for visually selecting sites for remote surgical guidance, characterized in that, A tracking method for performing the remote surgical guidance visual manual selection site as described in claim 1, including: A first acquisition module, configured to obtain the source point to be dynamically tracked as the target site according to the manual selection operation of the guiding doctor on the initial frame image, and the initial frame image is collected from the shared screen of the remote guiding end and the surgical end; A second acquisition module, configured to acquire a target region including the target site, perform a masking operation on the target region, detect SIFT feature points, and acquire a first set of feature points; Continuously track the target region, acquire a tracking region corresponding to each subsequent frame image of the initial frame image, perform a masking operation on the tracking region, detect SIFT feature points, and acquire a second set of feature points; A tracking module, configured to, according to the first set of feature points and the second set of feature points, obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination, and determine a homography matrix; A mapping module, configured to map the coordinate position of the target site by using the homography matrix to determine a tracking site corresponding to each subsequent frame image of the initial frame image.

4. The tracking system for visually manually selecting sites for remote surgical guidance according to claim 3, wherein, The tracking method further includes: An interaction module, configured to send the tracking site to a remote guidance end, and implement real-time communication between the doctor at the guidance end and the doctor at the surgical end through audio and video interaction.

5. A tracking device for visually manually selecting sites for remote surgical guidance, characterized in that, A tracking method for remotely guiding a visual artificial selection site in a surgical operation as claimed in claim 1, including a remote guidance end and a surgical end, where the remote guidance end includes a first interaction device, and the surgical end includes a second interaction device and a sampling device; The first interaction device is configured to receive the real-time surgical video collected by the sampling device as a shared screen between the remote guidance end and the surgical end, and obtain a source point that needs to be dynamically tracked as a target site according to the manual selection operation of the doctor at the guidance end on the initial frame image, where the initial frame image is collected from the shared screen; The second interaction device is configured to acquire a target region including the target site, perform a masking operation on the target region, detect SIFT feature points, and acquire a first set of feature points; And, continuously track the target region, acquire a tracking region corresponding to each subsequent frame image of the initial frame image, perform a masking operation on the tracking region, detect SIFT feature points, and acquire a second set of feature points; And, according to the first set of feature points and the second set of feature points, obtain the coordinate positions of the final matching points by performing feature matching and noise point elimination, and determine a homography matrix And, map the coordinate position of the target site by using the homography matrix to determine a tracking site corresponding to each subsequent frame image of the initial frame image, where the tracking site is used to guide the doctor at the surgical end to perform a surgical operation.

6. The tracking device for remotely guiding a visual artificial selection site in a surgical operation as claimed in claim 5, wherein The second interaction device is further configured to send the tracking site to the first interaction device; Both the first interaction device and the second interaction device include audio and video interaction devices; the audio and video interaction devices are configured to implement real-time communication between the doctor at the guidance end and the doctor at the surgical end.

Images