Method, device, terminal and readable storage medium for generating motion trajectory

Abstract

The application discloses a motion trajectory generation method and device, a terminal and a readable storage medium. The generation method comprises the following steps: obtaining the spatial position of an endoscope lens in image2 relative to image1 for any two continuous video frames image1 and image2 in the endoscope video; and obtaining the motion trajectory of the endoscope based on the position of the endoscope lens in the first video frame in the endoscope video and the spatial position of the endoscope lens in each of the remaining video frames relative to the previous video frame. The generation method can obtain the motion trajectory of the endoscope.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a method, apparatus, terminal, and readable storage medium for generating motion trajectories. Background Technology

[0002] Endoscopes are a new technological product in medical development, and they are increasingly being widely used in the clinical diagnosis of various diseases. They provide painless and non-invasive monitoring and diagnosis by entering the abdominal cavity (e.g., stomach, intestines, bronchi, and kidneys) and using their lens components to take close-up pictures of the condition of the abdominal cavity walls for clinical diagnosis, thereby reducing the clinical suffering of patients.

[0003] Once an endoscope is inserted into the body, it is necessary to take images of the abdominal cavity. The endoscope is in a free state, floating freely in the body fluid. Due to its uncertain position, the images taken are highly arbitrary. It is sometimes difficult to determine the location of the images, the overall condition of the abdominal cavity, or the location of the tumor. How to determine the location of the images in the abdominal cavity and thus obtain the trajectory of the endoscope has become an important problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the main objective of the present invention is to provide a method, apparatus, terminal and readable storage medium for generating motion trajectories.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A method for generating the motion trajectory of an endoscope in endoscopic video, comprising the following steps: performing the following processing on any two consecutive video frames image1 and image2 in the endoscopic video: obtaining the rotation angle α between image1 and image2, and obtaining the planar movement distance move between image1 and image2. x,y Obtain the forward and backward movement distance between image1 and image2. z Move the plane a distance x,y and the distance moved forward and backward z Distance mapped to the matching application scenario x,y and distance z The spatial position of the endoscope lens in image2 relative to image1 is obtained; based on the position of the endoscope lens in the first video frame of the endoscope video, and the spatial position of the endoscope lens in each of the remaining video frames relative to the previous video frame, the motion trajectory of the endoscope is obtained.

[0006] As an improvement of this embodiment of the invention, the step of "obtaining the rotation angle α between image1 and image2" specifically includes: performing Surf feature point extraction processing on both image1 and image2, and performing Flann feature point matching processing to obtain a number of matched pairs of first feature points, and calculating the angle difference for each matched pair of first feature points, where α = the average value of all angle differences.

[0007] As an improvement to this embodiment of the invention, the step of "obtaining the planar movement distance between image1 and image2" is described. x,y Specifically, this includes: performing optical flow tracking on images1 and 2 and extracting the coordinates of matched pairs of second feature points; calculating the coordinate difference for each matched pair of second feature points; and moving... x,y = Average of all coordinate differences * cosα.

[0008] As an improvement to an embodiment of the present invention, the step of “performing optical flow tracing on image1 and image2 and extracting the coordinates of the matching second feature points” specifically includes: performing optical flow tracing on image1 and image2 based on the Lucas-Kanade method and extracting the coordinates of the matching second feature points.

[0009] As an improvement to this embodiment of the invention, the step of "obtaining the forward and backward movement distance between image1 and image2" is described. z Specifically, this includes: generating a depth map depth_image1 corresponding to image1, generating a depth map depth_image2 corresponding to image2, obtaining the coordinates of each matched pair of second feature points respectively in the first depth value in depth_image1 and the second depth value in depth_image2, obtaining the depth difference between the second depth value and the first depth value corresponding to each second feature point, and moving... z = The average value of the depth difference.

[0010] As an improvement to an embodiment of the present invention, the step of "generating the depth map depth_image1 corresponding to image1 and generating the depth map depth_image2 corresponding to image2" specifically includes: generating the depth map depth_image1 corresponding to image1 and generating the depth map depth_image2 corresponding to image2 based on the CycleGAN neural network.

[0011] This invention also provides a device for generating the motion trajectory of an endoscope in endoscopic video, comprising the following modules: a processing module, configured to perform the following processing on any two consecutive video frames image1 and image2 in the endoscopic video: obtain the rotation angle α between image1 and image2, and obtain the planar movement distance move between image1 and image2. x,y Obtain the forward and backward movement distance between image1 and image2. z Move the plane a distance x,y and the distance moved forward and backward z Distance mapped to the matching application scenario x,y and distance z The endoscope lens in image2 is obtained relative to image1. The summarization module is used to obtain the motion trajectory of the endoscope based on the position of the endoscope lens in the first video frame of the endoscope video and the spatial position of the endoscope lens in each of the remaining video frames relative to the previous video frame.

[0012] As an improvement of this embodiment of the invention, the processing module is further configured to: perform Surf feature point extraction processing on both image1 and image2, and perform Flann feature point matching processing to obtain several matched pairs of first feature points, and calculate the angle difference for each matched pair of first feature points, where α = the average value of all angle differences.

[0013] This invention also provides a terminal, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the generation method described above.

[0014] This invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the generation method described above.

[0015] The motion trajectory generation method, apparatus, terminal, and readable storage medium provided in this invention have the following advantages: This invention discloses a motion trajectory generation method, apparatus, terminal, and readable storage medium. The generation method includes: obtaining the spatial position of the endoscope lens in image2 relative to image1 for any two consecutive video frames image1 and image2 in the endoscope video; and obtaining the motion trajectory of the endoscope based on the position of the endoscope lens in the first video frame and the spatial position of the endoscope lens in each of the remaining video frames relative to the previous video frame. This generation method can acquire the motion trajectory of the endoscope. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the method for generating motion trajectories in this embodiment. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0018] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0019] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings and are used only for the convenience of describing this document and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements, or direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0020] Embodiment 1 of the present invention provides a method for generating the motion trajectory of an endoscope for endoscopic video, such as... Figure 1 As shown, it includes the following steps:

[0021] Step 101: For any two consecutive video frames image1 and image2 in the endoscopic video, perform the following processing: obtain the rotation angle α between image1 and image2, and obtain the planar movement distance move between image1 and image2. x,y Obtain the forward and backward movement distance between image1 and image2. z Move the plane a distance x,y and the distance moved forward and backward z Distance mapped to the matching application scenario x,y and distance z This process yields the spatial position of the endoscope lens in image2 relative to image1. Here, the endoscopic video can be considered as consisting of multiple video frames arranged in chronological order. Therefore, the following processing is performed on every two consecutive video frames to obtain the spatial position of the endoscope lens in each video relative to the previous video frame. For ease of description, a three-dimensional space can be created; for example, the plane where each video frame is located is the XOY plane, and the extension direction of multiple consecutive video frames is the Z-axis, etc. Rotation angle combined with optical flow tracking can determine the movement of feature points on the plane (X-axis and Y-axis), and depth estimation can determine the movement of feature points on the Z-axis.

[0022] Step 102: Based on the position of the endoscope lens in the first video frame of the endoscopic video, and the spatial position of the endoscope lens relative to the previous video frame in each of the remaining video frames, the motion trajectory of the endoscope is obtained. Here, for the first video frame of the endoscopic video, it is only necessary to obtain the position of the endoscope lens in the first video frame. For the remaining video frames, the spatial position of the endoscope lens relative to the previous video frame can be obtained. Therefore, by superimposing all the video frames, the spatial position of the endoscope lens in each video frame relative to the first video frame can be obtained, and thus the motion trajectory of the endoscope can be obtained.

[0023] In this embodiment, "obtaining the rotation angle α between image1 and image2" specifically includes: performing SURF feature point extraction processing on both image1 and image2, and then performing Flann feature point matching processing to obtain several matched pairs of first feature points. For each matched pair of first feature points, an angle difference is calculated, where α = the average of all angle differences. Obtaining the rotation angle α ensures accurate orientation; for example, what was originally left becomes right after rotating 180 degrees. Without knowing the rotation angle, the orientation will be significantly off. It is understandable that using the above method to calculate the angle difference has advantages such as speed and stable feature point matching.

[0024] The Surf (Speeded Up Robust Feature) algorithm is an improved and accelerated version of the Sift algorithm, with better overall performance. The basic process of the Surf algorithm is as follows: Step 1, construct the Hessian matrix and generate interest points for feature extraction; Step 2, construct the scale space; Step 3, feature point localization; Step 4, assign principal directions of feature points; Step 5, generate feature point descriptors; Step 6, feature point matching.

[0025] Flann (Fast Library for Approximate Nearest Neighbors) feature point matching records the keypoints of the target image and the image to be matched. It then constructs a descriptor based on the keypoint set, compares and filters these descriptors, and finally obtains a set of mappings for matching points. The size of this set can also be used to measure the degree of matching between the two images.

[0026] Here, we first perform the following processing on each pair of Surf feature points that are matched: obtain the angle between the two Surf feature points, and then calculate the angle difference between the two angles.

[0027] α = mean(angle) featurePoint2-i -angle featurePoint1-i ), where mean() is the average function, angle featurePoint2-i Let angle be the angle of the i-th matching feature point in image2. featurePoint1-i Let i be the angle of the i-th matching feature point in image1, where i is a positive integer. It can be understood that when there are N matching Surf feature points, i takes the value 1, 2, ..., N.

[0028] In this embodiment, the step of "obtaining the planar movement distance between image1 and image2" is described. x,Specifically, this includes: performing optical flow tracking on images1 and 2 and extracting the coordinates of matched pairs of second feature points; calculating the coordinate difference for each matched pair of second feature points; and moving... x, = Average of all coordinate differences * cosα.

[0029] Here, we first perform the following processing on each pair of matching feature points: obtain the coordinates of the two feature points, and then calculate the difference between the two coordinates.

[0030] move x,y =mean(position) featurePoint2-i -position featurePoint1-i )*cosα, where mean() is the average function, and position featurePoint2-i Let position be the coordinate of the i-th matching feature point in image2. featurePoint1-i Let i be the coordinates of the i-th matching feature point in image1, where i is a positive integer. It can be understood that when there are M matching feature points, i takes the values ​​1, 2, ..., M.

[0031] In this embodiment, the step of "performing optical flow tracing on image1 and image2 and extracting the coordinates of the matching second feature points" specifically includes: performing optical flow tracing on image1 and image2 based on the Lucas-Kanade method and extracting the coordinates of the matching second feature points.

[0032] Here, the Lucas-Kanade method is a differential method for optical flow estimation. It assumes that the optical flow is constant in the neighborhood of a pixel, and then uses the least squares method to solve the basic optical flow equation for all pixels in the neighborhood. The Lucas-Kanade method offers advantages such as stable performance when used for feature point tracking.

[0033] In this embodiment, the step of "obtaining the forward and backward movement distance between image1 and image2" is described. z Specifically, this includes: generating a depth map depth_image1 corresponding to image1, generating a depth map depth_image2 corresponding to image2, obtaining the coordinates of each matched pair of second feature points respectively in the first depth value in depth_image1 and the second depth value in depth_image2, obtaining the depth difference between the second depth value and the first depth value corresponding to each second feature point, and moving... z = The average value of the depth difference.

[0034] Here, we first perform the following processing on each pair of matching feature points: obtain the depth of the two feature points, and then calculate the difference between the depth values ​​of these two coordinates. z =mean(depth) featurePoint2-i -depth featurePoint1-i ), where mean() is the average function, and depth featurePoint2-i Let depth be the depth value of the i-th matching feature point in image2. featur]ePoint1-i Let i be the depth value of the i-th matching feature point in image1, where i is a positive integer. It can be understood that when there are M matching feature points, i takes the values ​​1, 2, ..., M.

[0035] In this embodiment, the step of "generating the depth map depth_image1 corresponding to image1 and the depth map depth_image2 corresponding to image2" specifically includes: generating the depth map depth_image1 corresponding to image1 and generating the depth map depth_image2 corresponding to image2 based on the CycleGAN neural network. Here, the training data of CycleGAN does not need to be one-to-one, which can save a lot of time in data collection and data preprocessing.

[0036] Embodiment 2 of the present invention provides a device for generating the motion trajectory of an endoscope in endoscopic video, comprising the following modules:

[0037] The processing module is configured to perform the following processing on any two consecutive video frames image1 and image2 in the endoscopic video: obtain the rotation angle α between image1 and image2, and obtain the planar movement distance move between image1 and image2. x,y Obtain the forward and backward movement distance between image1 and image2. z Move the plane a distance x,y and the distance moved forward and backward z Distance mapped to the matching application scenario x,y and distance z This gives the spatial position of the endoscope lens in image2 relative to image1;

[0038] The summarization module is used to obtain the motion trajectory of the endoscope based on the position of the endoscope lens in the first video frame of the endoscope video and the spatial position of the endoscope lens relative to the previous video frame in each of the remaining video frames.

[0039] In this embodiment, the processing module is further configured to: perform Surf feature point extraction processing on both image1 and image2, and perform Flann feature point matching processing to obtain several matched pairs of first feature points, and calculate the angle difference for each matched pair of first feature points, where α = the average value of all angle differences.

[0040] Embodiment 3 of the present invention provides a terminal, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the generation method as described in Embodiment 1.

[0041] Embodiment 4 of the present invention provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the generation method as described in Embodiment 1.

[0042] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0043] This invention can be a system, method, and / or computer program product. A computer program product may include a readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0044] A readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. Readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0045] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating the motion trajectory of an endoscope in endoscopic video, characterized in that, Includes the following steps: For any two consecutive video frames image1 and image2 in the endoscopic video, the following processing is performed: the rotation angle α between image1 and image2 is obtained; optical flow tracing is performed on image1 and image2 based on the Lucas-Kanade method to extract the coordinates of the matched second feature points; the coordinate difference is calculated for each matched second feature point. =Average of all coordinate differences* ; Based on the CycleGAN neural network, a depth map depth_image1 corresponding to image1 and a depth map depth_image2 corresponding to image2 are generated. The coordinates of each matched pair of second feature points are obtained, namely the first depth value in depth_image1 and the second depth value in depth_image2. The depth difference between the second depth value and the first depth value corresponding to each second feature point is obtained. =The average value of the depth difference; the distance the plane is moved and forward and backward movement distance Distance mapped to matching application scenarios and This gives the spatial position of the endoscope lens in image2 relative to image1; The motion trajectory of the endoscope is obtained based on the position of the endoscope lens in the first video frame of the endoscope video, and the spatial position of the endoscope lens relative to the previous video frame in each of the remaining video frames.

2. The generation method according to claim 1, characterized in that, The phrase "obtaining the rotation angle α between image1 and image2" specifically includes: Both image1 and image2 are subjected to SURF feature point extraction and Flann feature point matching to obtain several matched pairs of first feature points. An angle difference is calculated for each matched pair of first feature points, where α = the average of all angle differences.

3. A device for generating the motion trajectory of an endoscope for endoscopic video, characterized in that, Includes the following modules: The processing module is used to perform the following processing on any two consecutive video frames image1 and image2 in the endoscopic video: obtain the rotation angle α between image1 and image2; perform optical flow tracing on image1 and image2 based on the Lucas-Kanade method and extract the coordinates of the matching pairs of second feature points; calculate the coordinate difference for each matching pair of second feature points. =Average of all coordinate differences* Based on the CycleGAN neural network, a depth map depth_image1 corresponding to image1 and a depth map depth_image2 corresponding to image2 are generated. The coordinates of each matched pair of second feature points are obtained, namely the first depth value in depth_image1 and the second depth value in depth_image2. The depth difference between the second depth value and the first depth value corresponding to each second feature point is obtained. =The average value of the depth difference; the distance the plane is moved and forward and backward movement distance Distance mapped to matching application scenarios and This gives the spatial position of the endoscope lens in image2 relative to image1; The aggregation module is used to obtain the motion trajectory of the endoscope based on the position of the endoscope lens in the first video frame of the endoscope video and the spatial position of the endoscope lens relative to the previous video frame in each of the remaining video frames.

4. The generating apparatus according to claim 3, characterized in that, The processing module is also used for: Both image1 and image2 are subjected to SURF feature point extraction and Flann feature point matching to obtain several matched pairs of first feature points. An angle difference is calculated for each matched pair of first feature points, where α = the average of all angle differences.

5. A terminal, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the generation method as described in claim 1 or 2.

6. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the generation method as described in claim 1 or 2.

Images