Video image fusion processing method and device for capsule endoscope

Abstract

The invention discloses a video image fusion processing method and device for a capsule endoscope, and the method comprises the steps: loading a plurality of segments of videos at different angles, which are shot by the capsule endoscope, into a processing module, carrying out the image enhancement preprocessing of the nth frame of each segment of video, carrying out the rough image registration of the preprocessed image through the pose data of a camera, and carrying out the image fusion processing; and carrying out image fine registration by using an SURF algorithm to determine an overlapping region, searching an optimal suture line in the overlapping region by using an optimal suture line image fusion algorithm, carrying out image stitching, and carrying out video stitching on the processed stitching image set. According to the invention, image coarse registration is carried out according to the pose data of the capsule endoscope, an SURF image registration algorithm is improved, and optimization and improvement are carried out in a feature point matching stage, so that the accuracy and timeliness of the algorithm are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, and in particular to a video image fusion processing method and device for capsule endoscope. Background Art

[0002] As a non-invasive endoscopic examination method, capsule endoscopy technology is of great significance in the field of medical imaging. The working principle of capsule endoscopy is that the patient takes a capsule orally, uses the built-in camera of the capsule to capture images of organs such as the digestive tract, and transmits them wirelessly to external devices for image display and analysis. However, since the capsule endoscope needs to be small in size, its shooting angle and field of view are limited. At present, many capsule endoscopy host computer systems are not equipped with corresponding video image stitching algorithms, and can only provide relatively limited single-view images, which limits the doctor's comprehensive assessment of the condition.

[0003] With the development of capsule endoscopy technology, its corresponding video stitching technology has also begun to receive attention. Video stitching technology relies on image stitching technology, which refers to stitching multiple images with overlapping areas into a wide-angle panoramic image according to specific rules. It mainly includes three steps: image preprocessing, image registration, and image fusion, among which registration and fusion are the two key steps of image stitching technology. At present, there are many feasible image matching stitching algorithms, but in actual operation, there may be problems such as excessive computation or slow operation speed, which affects real-time performance and efficiency. In addition, due to the special characteristics and limitations of images taken by capsule endoscopes, traditional stitching algorithms may not be well adapted to the application scenarios of capsule endoscopes, and need to be optimized and improved in a targeted manner. Summary of the invention

[0004] The purpose of the present invention is to solve the above-mentioned technical problems and provide a video image fusion processing method and device for capsule endoscope. The present invention performs coarse image registration based on the posture data of the capsule endoscope, improves the SURF image registration algorithm, and optimizes and improves the feature point matching stage, thereby improving the accuracy and timeliness of the algorithm.

[0005] The technical solution adopted by the present invention to solve the above technical problem is: a video image fusion processing method for capsule endoscopy, comprising the following steps:

[0006] Step 1: Load multiple videos from different angles captured by the camera of the capsule endoscope into the processing module;

[0007] Step 2: Take the nth frame of each video for image enhancement preprocessing;

[0008] Step 3: The preprocessed image is roughly registered using the camera pose data. The rough registration includes the following: projecting the image and calculating the intersection of the two images in the three-dimensional space, calculating the overlapping part of the projected image based on the intersection and the edge vertices of the projected image to obtain the shortest closed connected path, and inversely transforming the obtained shortest closed connected path to its position in the image to obtain the overlapping part of the image;

[0009] Step 4: Use the SURF algorithm to perform image registration to determine the overlapping area. The image registration includes the following: extract the feature points in the overlapping part of the image in step 3, use the rotation angle of the camera itself as the main direction and use the circular neighborhood of the feature point to calculate the feature point descriptor, perform dimensionality reduction on the calculated feature point descriptor, and then use bidirectional matching to determine the image overlapping area;

[0010] Step 5: Use the best stitching line image fusion algorithm to find the best stitching line in the overlapping area and perform image stitching;

[0011] Step 6: If n < the number of video frames, repeat steps 2 to 5; if n ≥ the number of video frames, perform video stitching on the processed stitched image set.

[0012] Preferably, the rough image registration in step 3 includes the following contents:

[0013] The image obtained after preprocessing is height*width, which defines the angular coordinates in the two-dimensional image plane coordinate system

[0014]

[0015] The projected image is at a distance d from the camera;

[0016] The position and posture of the capsule endoscope can be expressed by six degrees of freedom. The position and posture are expressed as two matrices, namely the position matrix With the attitude angle matrix u, v, w are the angle with the y-axis, the angle with the xOy plane, and the rotation angle of the camera itself respectively;

[0017] The unit vector of the camera's direction

[0018]

[0019] The image projection position is

[0020] I=P+d*OV;

[0021] The angular coordinate matrix of the projection plane in the spatial coordinate system can be calculated

[0022] CV1=OV×[0 0 1]; RV1=Rotate(CV1,OV,w);

[0023] CV2=OV×CV1; RV2=Rotate(CV2,OV,w);

[0024] Point 3d =I+x Point *RV1+y Point *RV2;

[0025] Among them, the rotation function Rotate is defined as follows:

[0026] Rotate(vector,axis,θ)=R*vector;

[0027]

[0028] If the projections of the two images have an intersection in the three-dimensional space, then we can traverse each edge of the two projections, calculate the intersection, calculate the overlapping part based on the intersection and the vertex, and find the shortest closed connected path on the edge of the projection plane that includes all the intersections;

[0029] All nodes of the calculated shortest closed connected path are inversely transformed to their positions in the image, that is, the overlapping parts of the image are found.

[0030] Preferably, the method of traversing each edge of the two projections to calculate the intersection point includes the following contents:

[0031] The two edges are l1 and l2, where the endpoints of l1 are (p1, p2); the endpoints of l2 are (q1, q2); calculate the direction vector of the edge: l1 direction vector l2 direction vector

[0032] Determine whether two edges are parallel or collinear:

[0033] cp1=v1×(q1-p1); cp2=v1×(q2-p1);

[0034] cp3=v2×(p1-q1); cp4=v2×(p2-q1);

[0035] If cp1·cp2≥0 or cp3·cp4≥0, it indicates parallelism or collinearity;

[0036] If the two sides are not parallel, the intersection point of the two sides can be calculated:

[0037]

[0038] If both t and s are within the range of [0, 1], there is an intersection point; otherwise, there is no intersection point.

[0039] The intersection point is p1 + t * v1 or q1 + s * v2.

[0040] Preferably, step three further includes the following: determining whether the line connecting any two points of the intersection point and / or vertex is on the plane edge, including the following steps:

[0041] Step 1: Generate a vector group vectors of the edges according to the vertex group vertices;

[0042] Step 2: Convert vectors into a unit vector group unit_vectors;

[0043] Step 3: Denote any two points of the intersection point and / or vertex as point1 and point2, calculate the vector v from point1 to point2 and the distance d, and obtain the unit vector v u ;

[0044] Step 4: Traverse unit_vectors to determine whether there is a vector u that matches v u , that is, whether it is on its edge. Only need: |u · v u -1| < t, where the tolerance (t) is a preset value, that is, the allowable range of floating-point precision error. By determining whether the line connecting any two points of the intersection point and / or vertex is on the plane edge, it is avoided that the path directly passes through the interior. Through this step setting, it is convenient for the path to pass through the plane edge.

[0045] Preferably, the calculation method for obtaining the shortest closed connected path by calculating the overlapping part of the projection image according to the intersection point and the edge vertices of the projection image in step three is as follows:

[0046] Search for the intersection points, and record the number as n; then all the intersection points and vertices are recorded as nodes, and the number of nodes N = n + 8;

[0047] Number all the nodes, where the intersection points are 1 to n, and the vertices are n + 1 to N;

[0048] Construct an adjacency matrix M, set the initial distance as Inf, and take the distance for any two points whose connection line is on the plane edge, and adjust the corresponding elements in M;

[0049] Construct an undirected graph G with M, and do not connect the lines if the distance is Inf;

[0050] Start traversing from node 1, and take the set S curr of the connected node of the currently traversed node P conn , and before changing P curr , set P last as P curr Divide into the following situations:

[0051] a) S conn When the number is 1, P curr Recorded in visited, P curr Set as the connection node;

[0052] b) If S conn If there is any intersection point in P curr Recorded in visited, P curr Set as the intersection point;

[0053] c) If P curr It is an intersection point and not the last intersection point P n , and the first two conditions are not met, then remove S conn P last , and then remove and P last With P curr The next intersection point of the connected lines is P. next =P curr +1, use the depth-first search DFS algorithm to find the paths where the remaining nodes are located, and select P next The shortest path, record the nodes in the path into visited, P curr Set to P next ;

[0054] d) If P curr is the last intersection, then the target node is set to 1, that is, the initial node P begin . Still use the DFS algorithm to search and select P begin The shortest path records the nodes in the path in visited.

[0055] According to the number in visited, the corresponding node coordinates are taken out and the shortest closed connected path is stored as path.

[0056] Preferably, in step 3, the shortest closed connected path obtained is reversely transformed to its position in the image to obtain the overlapping part of the image, which includes the following contents:

[0057] The three-dimensional space point coordinates are converted into two-dimensional image plane coordinates. The calculation method includes the following:

[0058] V p =Point 3d -P;

[0059] Distance=V p OV;

[0060] V proj =V p -Distance*OV;

[0061]

[0062] The closed path formed by the coordinate points is the overlapping part calculated based on the pose data, which realizes the rough registration of the image.

[0063] Preferably, converting the three-dimensional space point coordinates into the two-dimensional image plane coordinates also includes the following:

[0064] If the camera itself has a rotation angle, the coordinates must also be rotated to match the image coordinates. The calculation method includes the following:

[0065] Rotation Matrix

[0066]

[0067] Coordinates after rotation

[0068] Point 2d =(R P ×Point′ 2d )′.

[0069] Preferably, in step 4, the dimension reduction processing of the calculated feature point descriptor includes the following contents:

[0070] Draw another circle with a radius of r inside the circular neighborhood, take four sub-regions of quadrant 1-3, and mark them 1 to 4 from the outside to the inside in a counterclockwise direction starting from the first quadrant;

[0071] Calculate the Haar wavelet response value of the feature point in each sub-region in the x-direction and y-direction, and weight it by the Gaussian function, then count the sum of the response values ​​in the x-direction and y-direction of each sub-region and the sum of the absolute values ​​of the response values ​​in the x-direction and y-direction, and you can get a 4-dimensional feature point descriptor

[0072] V=(∑dx,∑|dx|,∑dy,∑|dy|);

[0073] By counting the response values ​​of the four sub-regions, we can get a 16-dimensional feature point descriptor, and the calculated feature vector can be expressed as

[0074]

[0075] Preferably, the two-way matching in step 4 includes the following contents:

[0076] Obtain the feature points of the reference and matching images, denoted as sets p and q respectively;

[0077] Traverse all feature points in set p and search for corresponding matching points in set q, recorded as set {i:p→q};

[0078] Traverse all feature points in set q and search for corresponding matching points in set p, recorded as set {j:p→q};

[0079] Compare all matching point pairs in sets i and j, and retain the mapping relationships of the same elements as correct feature point pairs, otherwise delete them.

[0080] Preferably, step S is further included between step 1 and step 2, and step S includes the following contents: if the video frame numbers are different, the video is normalized. The number of video frames after normalization is the number of frames of the shortest video in the video set to ensure subsequent processability.

[0081] A video image fusion processing device for capsule endoscope comprises a processing module, wherein the processing module is provided with a program of the video image fusion processing method for capsule endoscope.

[0082] The present invention has the following beneficial effects:

[0083] 1. The video image fusion processing algorithm implemented by the present invention can solve the problem that the video frame taken by the capsule endoscope is too small. By improving the algorithm, the spliced ​​video is relatively more natural and complete, providing clear image data;

[0084] 2. The present invention performs rough image registration based on the position data of the capsule endoscope, improves the SURF image registration algorithm, and optimizes and improves the feature point matching stage, thereby improving the accuracy and timeliness of the algorithm;

[0085] 3. The traditional SURF algorithm mainly includes five steps: constructing scale space, locating feature points, allocating main directions of feature points, generating feature point descriptors, and matching feature points. In the step of generating feature point descriptors, the traditional SURF algorithm often uses square neighborhoods of feature points to generate feature point descriptors. In the present invention, the image direction is determined according to the camera pose data, so the main direction allocation can be ignored, and the rotation angle w of the camera itself is directly used as the main direction, and the circular neighborhood of the feature points is used to calculate the feature point descriptors; 2.

[0086] In the traditional SURF algorithm, the square neighborhood of the feature point is divided into 16 uniform square sub-areas, resulting in 64 dimensions. Since the present invention uses coarse alignment, the dimension of the feature point descriptor can be further reduced. Compared with the 64-dimensional feature descriptor of the traditional SURF algorithm, the feature descriptor of the present invention only needs 16 dimensions. Compared with the original algorithm, further dimensionality reduction processing is performed to reduce the algorithm complexity and optimize the algorithm running speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a schematic diagram of the process of the present invention.

[0088] Figure 2 It is a schematic diagram of three-dimensional space simulation projection.

[0089] Figure 3 It is a schematic diagram of overlapping area calculation.

[0090] Figure 4 It is a schematic diagram of the reverse conversion of three-dimensional coordinates into two-dimensional coordinates on the image.

[0091] Figure 5 This is a comparison chart between a square area and a circular area.

[0092] Figure 6 It is a schematic diagram of circular neighborhood division.

[0093] Figure 7 is a schematic diagram of the optimal suture line. DETAILED DESCRIPTION

[0094] The present invention is further described below in conjunction with the accompanying drawings and implementation modes.

[0095] like Figure 1-7 As shown, the present invention provides a method for video image fusion processing for capsule endoscopy, comprising the following steps:

[0096] Step 1: Load multiple videos from different angles captured by the camera of the capsule endoscope into the processing module;

[0097] Step S: If the video frame numbers are the same, start processing; if the video frame numbers are different, normalize the videos, and the frame number of the normalized video is the frame number of the shortest video in the video set to ensure subsequent processability;

[0098] Step 2: Take the nth frame of each video and perform image enhancement preprocessing; the preprocessing includes noise reduction, brightness adjustment, and sharpening enhancement to facilitate image enhancement processing for subsequent algorithm operations. These operations already have corresponding excellent algorithms and can be directly called without optimization;

[0099] Step 3: The preprocessed image is roughly registered using the camera pose data. The rough registration includes the following: projecting the image and calculating the intersection of the two images in the three-dimensional space, calculating the overlapping part of the projected image based on the intersection and the edge vertices of the projected image to obtain the shortest closed connected path, and inversely transforming the obtained shortest closed connected path to its position in the image to obtain the overlapping part of the image;

[0100] like Figure 2 As shown, the image coarse registration includes the following:

[0101] The image obtained after preprocessing is height*width, which defines the angular coordinates in the two-dimensional image plane coordinate system

[0102]

[0103] The projected image is at a distance d from the camera;

[0104] The position and posture of the capsule endoscope can be expressed by six degrees of freedom. The position and posture are expressed as two matrices, namely the position matrix With the attitude angle matrix u, v, w are the angle with the y-axis, the angle with the xOy plane, and the rotation angle of the camera itself respectively;

[0105] The unit vector of the camera's direction

[0106]

[0107] The image projection position is

[0108] I=P+d*OV;

[0109] The angular coordinate matrix of the projection plane in the spatial coordinate system can be calculated

[0110] CV1=OV×[0 0 1]; RV1=Rotate(CV1,OV,w);

[0111] CV2=OV×CV1; RV2=Rotate(CV2,OV,w);

[0112] Point 3d =I+x Point *RV1+y Point *RV2;

[0113] Among them, the rotation function Rotate is defined as follows:

[0114] Rotate(vector,axis,θ)=R*vector;

[0115]

[0116] If the two image projections have an intersection in three-dimensional space, we can traverse each edge of the two projections, calculate the intersection, calculate the overlapping part based on the intersection and the vertex, and find the shortest closed connected path on the edge of the projection plane that contains all the intersections, such as Figure 3 As shown;

[0117] Reversely transform all nodes of the calculated shortest closed connected path path to their positions in the image, that is, find the overlapping part of the image;

[0118] The method of traversing each edge of the two projections to calculate the intersection point includes the following:

[0119] The two edges are l1 and l2, where the endpoints of l1 are (p1, p2); the endpoints of l2 are (q1, q2); calculate the direction vector of the edge: l1 direction vector l2 direction vector

[0120] Determine whether two edges are parallel or collinear:

[0121] cp1=v1×(q1-p1); cp2=v1×(q2-p1);

[0122] cp3=v2×(p1-q1); cp4=v2×(p2-q1);

[0123] If cp1·cp2≥0 or cp3·cp4≥0, it indicates parallelism or collinearity;

[0124] If the two sides are not parallel, the intersection point of the two sides can be calculated:

[0125]

[0126] If t and s are both in the range [0,1], there is an intersection, otherwise there is no intersection.

[0127] The intersection point is p1+t*v1 or q1+s*v2.

[0128] Determining whether the intersection point and / or the line connecting any two vertex points is on the edge of the plane includes the following steps:

[0129] Step 1: Generate edge vector groups vectors based on vertex groups vertices;

[0130] Step 2: Convert vectors to unit vector group unit_vectors;

[0131] Step 3: Let any two points of intersection and / or vertex be called point1 and point2, calculate the vector v and distance d from point1 to point2, and find the unit vector v u ;

[0132] Step 4: Traverse unit_vectors to determine whether there is a u The matching vector u, that is, whether it is on its edge.

[0133] The calculation method for calculating the shortest closed connected path by calculating the overlapping part of the projected image based on the intersection points and the edge vertices of the projected image is as follows:

[0134] Find the intersection points, the number is recorded as n; then all the intersection points and vertices are recorded as nodes, the number of nodes N = n + 8;

[0135] Number all nodes, where intersections are 1 to n and vertices are n+1 to N;

[0136] Construct the adjacency matrix M, set the initial distance to Inf, take the distance between any two points on the edge of the plane, and adjust the corresponding elements in M;

[0137] Construct an undirected graph G with M. If the distance is Inf, no connection is made;

[0138] Start traversing from node 1 and take the current traversal node P curr The set of connected nodes S conn , when changing P curr Former P last Set to P curr The following situations:

[0139] a)S conn When the number is 1, P curr Recorded in visited, P curr Set as the connection node;

[0140] b) If S conn If there is any intersection point in P curr Recorded in visited, P curr Set as the intersection point;

[0141] c) If P curr It is an intersection point and not the last intersection point P n , and the first two conditions are not met, then remove S conn P last , and then remove and P last With P curr The next intersection point of the nodes where the lines are collinear is P next =P curr +1, use the depth-first search DFS algorithm to find the paths where the remaining nodes are located, and select P next The shortest path, record the nodes in the path into visited, P curr Set to P next ;

[0142] d) If P curr is the last intersection, then the target node is set to 1, which is the initial node P begin . Still use the DFS algorithm to search and select P begin The shortest path records the nodes in the path in visited.

[0143] According to the number in visited, the corresponding node coordinates are taken out and the shortest closed connected path is stored as path.

[0144] like Figure 4As shown, the shortest closed connected path is inversely transformed to its position in the image to obtain the overlapping part of the image, including the following:

[0145] The three-dimensional space point coordinates are converted into two-dimensional image plane coordinates. The calculation method includes the following:

[0146] V p =Point 3d -P;

[0147] Distance=V p OV;

[0148] V proj =V p -Distance*OV;

[0149]

[0150] If the camera itself has a rotation angle, the coordinates must also be rotated to match the image coordinates. The calculation method includes the following:

[0151] Rotation Matrix

[0152]

[0153] Coordinates after rotation

[0154] Point 2d =(R P ×Point′ 2d )′;

[0155] The closed path formed by the coordinate points is the overlapping part calculated based on the pose data, which realizes the rough registration of the image.

[0156] Step 4: Use the SURF algorithm to perform image registration to determine the overlapping area. The image registration includes the following: extract the feature points in the overlapping part of the image in step 3, use the rotation angle of the camera itself as the main direction and use the circular neighborhood of the feature point to calculate the feature point descriptor, perform dimensionality reduction on the calculated feature point descriptor, and then use bidirectional matching to determine the image overlapping area;

[0157] The dimension reduction process of the calculated feature point descriptor includes the following:

[0158] like Figure 5 and Figure 6 As shown, draw another circle with a radius of The circle is divided into four sub-areas of quadrant 1 and 3, starting from the first quadrant and marking them 1 to 4 in a counterclockwise direction from the outside to the inside;

[0159] Calculate the Haar wavelet response value of the feature point in each sub-region in the x-direction and y-direction, and weight it by the Gaussian function, then count the sum of the response values ​​in the x-direction and y-direction of each sub-region and the sum of the absolute values ​​of the response values ​​in the x-direction and y-direction, and you can get a 4-dimensional feature point descriptor

[0160] V=(∑dx,∑|dx|,∑dy,∑|dy|);

[0161] By counting the response values ​​of the four sub-regions, we can get a 16-dimensional feature point descriptor, and the calculated feature vector can be expressed as

[0162]

[0163] Bidirectional matching includes the following:

[0164] Obtain the feature points of the reference and matching images, denoted as sets p and q respectively;

[0165] Traverse all feature points in set p and search for corresponding matching points in set q, recorded as set {i:p→q};

[0166] Traverse all feature points in set q and search for corresponding matching points in set p, recorded as set {j:p→q};

[0167] Compare all matching point pairs in sets i and j, and retain the mapping relationships of the same elements as correct feature point pairs, otherwise delete them.

[0168] like Figure 7 As shown, step five: use the best stitching line image fusion algorithm to find the best stitching line in the overlapping area for image stitching;

[0169] Step 6: If n < the number of video frames, repeat steps 2 to 5; if n ≥ the number of video frames, perform video stitching on the processed spliced ​​image set. In the video stitching, the compression coding compensates by using the motion information between video frames, thereby reducing the jitter effect of the picture to ensure that the generated video picture is stable and clear.

[0170] Step 7: Output the obtained stitched video to the specified source through the program for easy observation.

[0171] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

Claims

1. A video image fusion processing method for capsule endoscopy, characterized in that: The following steps are involved: Step 1: Load multiple videos from different angles captured by the camera of the capsule endoscope into the processing module; Step 2: Take the nth frame of each video for image enhancement preprocessing; Step 3: The preprocessed image is roughly registered using the camera pose data. The rough registration includes the following: projecting the image and calculating the intersection of the two images in the three-dimensional space, calculating the overlapping part of the projected image based on the intersection and the edge vertices of the projected image to obtain the shortest closed connected path, and inversely transforming the obtained shortest closed connected path to its position in the image to obtain the overlapping part of the image; Step 4: Use the SURF algorithm to perform image registration to determine the overlapping area. The image registration includes the following: extract the feature points in the overlapping part of the image in step 3, use the rotation angle of the camera itself as the main direction and use the circular neighborhood of the feature point to calculate the feature point descriptor, perform dimensionality reduction on the calculated feature point descriptor, and then use bidirectional matching to determine the image overlapping area; Step 5: Use the best stitching line image fusion algorithm to find the best stitching line in the overlapping area and perform image stitching; Step 6: If n < the number of video frames, repeat steps 2 to 5; if n ≥ the number of video frames, perform video stitching on the processed stitched image set.

2. The video image fusion processing method for capsule endoscopy according to claim 1, characterized in that: The image coarse registration in step 3 includes the following: The image obtained after preprocessing is height*width, which defines the angular coordinates in the two-dimensional image plane coordinate system The projected image is at a distance d from the camera; The position and posture of the capsule endoscope can be expressed by six degrees of freedom. The position and posture are expressed as two matrices, namely the position matrix With the attitude angle matrix u, v, w are the angle with the y-axis, the angle with the xOy plane, and the rotation angle of the camera itself respectively; The unit vector of the camera's direction Then the image projection position is I = P + d * OV; The angular coordinate matrix of the projection plane in the spatial coordinate system can be calculated CV1=OV×[0 0 1]; RV1=Rotate(CV1,OV,w); CV2=OV×CV1; RV2=Rotate(CV2,OV,w); Point 3d =I+x Point *RV1+y Point *RV2; Among them, the rotation function Rotate is defined as follows: Rotate(vector,axis,θ)=R*vector; If the projections of the two images have an intersection in the three-dimensional space, then we can traverse each edge of the two projections, calculate the intersection, calculate the overlapping part based on the intersection and the vertex, and find the shortest closed connected path on the edge of the projection plane that includes all the intersections; All nodes of the calculated shortest closed connected path are inversely transformed to their positions in the image, that is, the overlapping parts of the image are found.

3. The video image fusion processing method for capsule endoscopy according to claim 2 is characterized in that: The method of traversing each edge of the two projections to calculate the intersection point includes the following: The two edges are l1 and l2, where the endpoints of l1 are (p1, p2); the endpoints of l2 are (q1, q2); calculate the direction vector of the edge: l1 direction vector l2 direction vector Determine whether two edges are parallel or collinear: cp1=v1×(q1-p1); cp2=v1×(q2-p1); cp3=v2×(p1-q1); cp4=v2×(p2-q1); If cp1·cp2≥0 or cp3·cp4≥0, it indicates parallelism or collinearity; If the two sides are not parallel, the intersection point of the two sides can be calculated: If t and s are both in the range [0,1], there is an intersection, otherwise there is no intersection. The intersection point is p1+t*v1 or q1+s*v2.

4. The video image fusion processing method for capsule endoscopy according to claim 3 is characterized in that: Step 3 also includes the following: Determining whether the intersection point and / or the line connecting any two vertex points is on the edge of the plane includes the following steps: Step 1: Generate edge vector groups vectors based on vertex groups vertices; Step 2: Convert vectors to unit vector group unit_vectors; Step 3: Let any two points of intersection and / or vertex be called point1 and point2, calculate the vector v and distance d from point1 to point2, and find the unit vector v u ; Step 4: Traverse unit_vectors to determine whether there is a u The matching vector u, that is, whether it is on its edge.

5. The video image fusion processing method for capsule endoscopy according to claim 4 is characterized in that: In step 3, the calculation method for calculating the overlapping part of the projected image based on the intersection point and the edge vertices of the projected image to obtain the shortest closed connected path is as follows: Find the intersection points, the number is recorded as n; then all the intersection points and vertices are recorded as nodes, the number of nodes N = n + 8; Number all nodes, where intersections are 1 to n and vertices are n+1 to N; Construct the adjacency matrix M, set the initial distance to Inf, take the distance between any two points on the edge of the plane, and adjust the corresponding elements in M; Construct an undirected graph G with M. If the distance is Inf, no connection is made; Start traversing from node 1 and take the current traversal node P curr The set of connected nodes S conn , when changing P curr Former P last Set to P curr The following situations: a)S conn When the number is 1, P curr Recorded in visited, P curr Set as the connection node; b) If S conn If there is any intersection point in P curr Recorded in visited, P curr Set as the intersection point; c) If P curr It is an intersection point and not the last intersection point P n , and the first two conditions are not met, then remove S conn P last , and then remove and P last With P curr The next intersection point of the nodes where the lines are collinear is P next =P curr +1, use the depth-first search DFS algorithm to find the paths where the remaining nodes are located, and select P next The shortest path, record the nodes in the path into visited, P curr Set to P next ; d) If P curr is the last intersection, then the target node is set to 1, that is, the initial node P begin . Still use the DFS algorithm to search and select P begin The shortest path records the nodes in the path in visited. According to the number in vistaed, the corresponding node coordinates are taken out and the shortest closed connected path is stored as path.

6. The video image fusion processing method for capsule endoscopy according to claim 1, characterized in that: In step 3, the shortest closed connected path obtained is reversely transformed to its position in the image to obtain the overlapping part of the image, including the following: The three-dimensional space point coordinates are converted into two-dimensional image plane coordinates. The calculation method includes the following: V p =Point 3d -P; Distance=V p ·OV; In proj =V p -Distance*OV; The closed path formed by the coordinate points is the overlapping part calculated based on the pose data, which realizes the rough registration of the image.

7. The method for video image fusion processing for capsule endoscopy according to claim 6, characterized in that: Converting the coordinates of a 3D space point to the coordinates of a 2D image plane also includes the following: If the camera itself has a rotation angle, the coordinates must also be rotated to match the image coordinates. The calculation method includes the following: Rotation Matrix Coordinates after rotation Point 2d =(RP×Point′ 2d )′。 8. The video image fusion processing method for capsule endoscopy according to claim 1, characterized in that: In step 4, the dimension reduction processing of the calculated feature point descriptors includes the following: Draw another circle with a radius of r inside the circular neighborhood, take four sub-regions of quadrant 1-3, and mark them 1 to 4 from the outside to the inside in a counterclockwise direction starting from the first quadrant; Calculate the Haar wavelet response value of the feature point in each sub-region in the x-direction and y-direction, and weight it by the Gaussian function, then count the sum of the response values ​​in the x-direction and y-direction of each sub-region and the sum of the absolute values ​​of the response values ​​in the x-direction and y-direction, and you can get a 4-dimensional feature point descriptor V=(∑dx,∑|dx|,∑dy,∑|dy|); By counting the response values ​​of the four sub-regions, we can get a 16-dimensional feature point descriptor, and the calculated feature vector can be expressed as 9. The video image fusion processing method for capsule endoscopy according to claim 1, characterized in that: The two-way matching in step 4 includes the following: Obtain the feature points of the reference and matching images, denoted as sets p and q respectively; Traverse all feature points in set p and search for corresponding matching points in set q, recorded as set {i:p→q}; Traverse all feature points in set q and search for corresponding matching points in set p, recorded as set {j:p→q}; Compare all matching point pairs in sets i and j, and retain the mapping relationships of the same elements as correct feature point pairs, otherwise delete them.

10. A video image fusion processing device for capsule endoscope, characterized in that: The method comprises a processing module, wherein the processing module is provided with a program of any video image fusion processing method for capsule endoscope as claimed in claims 1 to 9.

Images