Method and system for determining key frames of capsule endoscopy image sequence

Abstract

The present invention discloses a method and system for determining key frames of capsule endoscope image sequences. The present invention performs mirror processing on the traversed current image and the next frame image of the current image, performs position matching on each frame image in the processed image pair according to a preset SiameseCF network model, determines image similarity according to the obtained position matching result, and determines key frame images in the capsule endoscope image sequence according to the obtained similarities of all images at the end of the traversal. The present invention performs position matching on each frame image in the mirrored image pair according to a preset SiameseCF network model, thereby determining the image similarity between the current image and the next frame image of the current image. Compared with the prior art of screening and selecting films through manual intervention, which results in a long film reading period and low work efficiency, the present invention effectively reduces the film reading time and improves work efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method and system for determining key frames of a capsule endoscope image sequence. Background Art

[0002] Wireless Capsule Endoscopy (WCE) is a new type of non-invasive gastrointestinal examination tool, referred to as capsule endoscopy or capsule endoscopy. Clinically, after the patient swallows the capsule endoscopy into the human body, it moves forward under its own gravity and the peristalsis of the gastrointestinal tract, and the capsule endoscopy imaging system automatically captures the digestive tract scene at a speed of two frames per second. The captured image data is transmitted to the external receiving terminal in a wireless transmission manner. During the average 7-8 hours of digestive tract examination, tens of thousands of color digestive tract scene images can be collected.

[0003] Due to the shooting speed of capsule endoscopy and the peristaltic characteristics of the digestive tract scene, the number of image sequences collected for a single case is huge, and there are a large number of redundant images with extremely high similarity. Therefore, even for experienced clinicians, it takes 3-5 hours of concentrated attention to complete an accurate and correct reading of a single case to screen out images containing suspicious lesions from capsule endoscopy image data. This is an arduous and rather boring task.

[0004] Therefore, studying the similarity of capsule endoscopy images is of great significance to improving doctors' work efficiency. In the prior art, these images are screened out or marked from the capsule endoscopy image sequence through manual intervention, resulting in a long reading period and low work efficiency.

[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present invention is to provide a method and system for determining key frames of capsule endoscopy image sequences, aiming to solve the technical problems in the prior art of manual intervention in screening and film selection, resulting in long film reading period and low work efficiency.

[0007] To achieve the above object, the present invention provides a method for determining a key frame of a capsule endoscopy image sequence, the method comprising the following steps:

[0008] Traversing images in a capsule endoscopy image sequence, and acquiring a current image traversed and a next frame image of the current image;

[0009] Performing mirror processing on the current image and the next frame of the current image respectively to obtain a processed image pair;

[0010] Performing position matching on each frame of the processed image pair according to a preset SiameseCF network model to obtain a position matching result;

[0011] Determining the image similarity between the current image and the next frame image of the current image according to the position matching result;

[0012] At the end of the traversal, the key frame image in the capsule endoscopy image sequence is determined according to the obtained similarities of all images.

[0013] Optionally, the step of performing position matching on each frame of the processed image pair according to a preset SiameseCF network model to obtain a position matching result includes:

[0014] Using each current image in the processed image pair as a reference image and the next image of the reference image as a target image;

[0015] extracting an image block of a preset size from the reference image based on a preset interval;

[0016] Inputting the extracted image block and the target image into a preset SiameseCF network model for feature extraction to obtain a feature extraction result;

[0017] Determine the optimal offset positions of all image blocks in the reference image in the target image according to the feature extraction result;

[0018] A position matching result is determined according to the optimal offset position.

[0019] Optionally, the step of determining the image similarity between the current image and a next frame image of the current image according to the position matching result comprises:

[0020] determining an image matching block between the reference image and the target image according to the optimal offset position;

[0021] The position distribution similarity and texture similarity of the image block are determined by an integrated descriptor according to the image matching block, and the image similarity between the reference image and the target image is determined according to the position distribution similarity and texture similarity of the image block.

[0022] Optionally, the step of determining the position distribution similarity and texture similarity of the image block by an integrated descriptor according to the image matching block, and determining the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image block comprises:

[0023] Calculating a joint probability between the reference image and the target image based on a preset joint probability formula and the image matching block;

[0024] The position distribution similarity and texture similarity of the image blocks are determined according to the joint probability and the integrated descriptor, and the image similarity between the reference image and the target image is determined according to the position distribution similarity and texture similarity of the image blocks.

[0025] Optionally, the step of determining the position distribution similarity and texture similarity of the image blocks according to the joint probability and the integrated descriptor, and determining the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image blocks comprises:

[0026] If the feature descriptor vectors of each image block of the reference image and the feature descriptor vectors of each image block of the target image are independent of each other, the similarity between the feature descriptor vectors corresponding to the image matching blocks is calculated by integrating the descriptor and the joint probability;

[0027] If the positions of the image blocks in the reference image and the positions of the image blocks in the target image are independent of each other, calculating the position similarity of the image matching blocks by integrating the descriptor and the joint probability;

[0028] The image similarity between the reference image and the target image is determined according to the similarity between the feature descriptor vectors, the position similarity and a preset similarity threshold.

[0029] Optionally, before the step of traversing the images in the capsule endoscopy image sequence and acquiring the traversed current image and the next frame image of the current image, the step further includes:

[0030] Performing a circular mirror operation on each frame of the historical capsule endoscopy image sequence to obtain a mirrored capsule endoscopy image sequence;

[0031] Traversing the mirrored capsule endoscopy image sequence, taking two adjacent similar images traversed as a training sample pair, and generating a sample set according to the training sample pair;

[0032] An initial SiameseCF network model is trained according to the sample set to obtain a preset SiameseCF network model.

[0033] Optionally, the step of training the original SiameseCF network model according to the sample set to obtain a preset SiameseCF network model includes:

[0034] Using the central image block of each current image in the sample pair as a template image, and the next frame of the current image as a search image;

[0035] Inputting the template image and the search image into the original SiameseCF network model to obtain a score map of the template image on the search image;

[0036] The loss of the score graph is calculated by a preset label value, and the network parameters of the original SiameseCF network model are trained using a preset stochastic gradient descent method and the loss to obtain a preset SiameseCF network model.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a capsule endoscopy image sequence key frame determination system, the capsule endoscopy image sequence key frame determination system comprising:

[0038] An image traversal module is used to traverse images in a capsule endoscopy image sequence and obtain a current image traversed and a next frame image of the current image;

[0039] A mirror processing module, used to perform mirror processing on the current image and the next frame image of the current image respectively to obtain a processed image pair;

[0040] A feature extraction module is used to perform position matching on each frame of the processed image pair according to a preset SiameseCF network model to obtain a position matching result;

[0041] A similarity determination module, used to determine the image similarity between the current image and the next frame image of the current image according to the position matching result;

[0042] The key frame image determination module is used to determine the key frame image in the capsule endoscopy image sequence according to the obtained similarities of all images at the end of the traversal.

[0043] Optionally, the feature extraction module is further used to use each current image frame in the processed image pair as a reference image and the next image frame of the reference image as a target image;

[0044] The feature extraction module is further used to extract image blocks of a preset size from the reference image based on a preset interval;

[0045] The feature extraction module is further used to input the extracted image block and the target image into a preset SiameseCF network model for feature extraction to obtain a feature extraction result;

[0046] The feature extraction module is further used to determine the optimal offset positions of all image blocks in the reference image in the target image according to the feature extraction result;

[0047] The feature extraction module is further used to determine a position matching result according to the optimal offset position.

[0048] Optionally, the similarity determination module is further used to determine an image matching block between the reference image and the target image according to the optimal offset position;

[0049] The similarity determination module is further used to determine the position distribution similarity and texture similarity of the image block through the integrated descriptor according to the image matching block, and determine the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image block.

[0050] The present invention traverses the images in the capsule endoscope image sequence, and obtains the traversed current image and the next frame image of the current image; performs mirror processing on the current image and the next frame image of the current image respectively to obtain a processed image pair; performs position matching on each frame image in the processed image pair according to a preset SiameseCF network model to obtain a position matching result; determines the image similarity between the current image and the next frame image of the current image according to the position matching result; at the end of the traversal, determines the key frame image in the capsule endoscope image sequence according to the similarities of all the images obtained. Since the present invention performs position matching on each frame image in the mirrored image pair through a preset SiameseCF network model, thereby determining the image similarity between the current image and the next frame image of the current image, compared with the prior art of screening and selecting films through manual intervention, resulting in a long film reading time period and low work efficiency, the present invention effectively reduces the film reading time and improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of a first embodiment of a method for determining key frames of a capsule endoscopy image sequence according to the present invention;

[0052] Figure 2 A mirror image schematic diagram of a capsule endoscopy image according to a first embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention;

[0053] Figure 3 A schematic diagram of the workflow of the capsule endoscopy image sequence similarity based on the SiameseCF network in the first embodiment of the method for determining the key frames of the capsule endoscopy image sequence of the present invention;

[0054] Figure 4 It is a flow chart of a second embodiment of a method for determining key frames of a capsule endoscopy image sequence according to the present invention;

[0055] Figure 5 A schematic diagram of a capsule endoscopy image of extracting 36 image blocks according to the second embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention;

[0056] Figure 6 A schematic diagram of the SiameseCF network structure of the second embodiment of the method for determining key frames of capsule endoscopy image sequences of the present invention;

[0057] Figure 7 A schematic diagram of the ConvNet network structure of the second embodiment of the method for determining key frames of capsule endoscopy image sequences of the present invention;

[0058] Figure 8 A schematic diagram of convolution operation of a second embodiment of a method for determining key frames of a capsule endoscopy image sequence according to the present invention;

[0059] Fig. 9 A schematic diagram of the optimal offset of the second embodiment of the method for determining key frames of capsule endoscopy image sequences of the present invention;

[0060] Fig.10 A schematic diagram of matching two capsule endoscopy image blocks according to a second embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention;

[0061] Fig.11 A schematic diagram of integrated descriptor similarity evaluation corresponding to some image blocks of the second embodiment of the method for determining key frames of capsule endoscopy image sequences of the present invention;

[0062] Fig.12 It is a schematic flow chart of a third embodiment of a method for determining key frames of a capsule endoscopy image sequence according to the present invention;

[0063] Fig.13 A schematic diagram of training sample pairs according to a third embodiment of a method for determining key frames of a capsule endoscopy image sequence according to the present invention;

[0064] Fig.14 This is a structural block diagram of the first embodiment of the capsule endoscopy image sequence key frame determination system of the present invention.

[0065] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0067] Reference Figure 1 , Figure 1The flowchart of the first embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention is shown, and the first embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention is proposed.

[0068] In this embodiment, the method for determining key frames of capsule endoscopy image sequence comprises the following steps:

[0069] Step S10: traverse the images in the capsule endoscopy image sequence, and obtain the traversed current image and the next frame image of the current image.

[0070] It should be noted that the execution subject of this embodiment may be a capsule endoscope image sequence key frame determination device, such as: a device connected to a wireless capsule endoscope, the device may be a computer, a notebook, etc., and may also be other capsule endoscope image sequence key frame determination devices that can achieve the same or similar functions. This embodiment does not limit this. Here, the above-mentioned capsule endoscope image sequence key frame determination device will be taken as an example to illustrate this embodiment and the following embodiments.

[0071] It is understandable that the capsule endoscopy image sequence refers to the image sequence generated by the capsule endoscopy imaging system automatically photographing the digestive tract scene at a speed of two frames per second, wherein the capsule endoscopy imaging system is not limited to generating an image sequence at a speed of two frames per second, and can be specifically set according to actual conditions, and the image sequence is sorted in sequence according to the shooting time and stored in a preset database. The current image may refer to the first frame of the image in the image sequence sorted in sequence according to the shooting time, that is, the next frame of the current image refers to the next frame of the image that is ranked after the first frame according to the shooting time. For example: there are two images A and B. According to the shooting time, image A is ranked before image B, that is, when traversing the images, image A is used as the current image and image B is used as the next frame.

[0072] Step S20: mirroring the current image and the next frame of the current image to obtain a processed image pair.

[0073] It should be noted that the mirror processing refers to the processing operation of mapping pixels between the current image and the part of the next frame of the current image that does not contain image information. For example, for further explanation, please refer to Figure 2 Mirror image of capsule endoscopy image. An original capsule endoscopy image is shown in Figure 2As shown in (a), there are some black areas around it, which have a certain adverse effect on the matching between subsequent image blocks. Therefore, mirroring a capsule endoscopy image is beneficial to the subsequent work. For a 240×240 pixel capsule endoscopy image, the radius of the circular area with image information is about 105 pixels. In this embodiment, the radius parameter is not limited to 105 pixels, but can also be other parameters. Based on this, the pixel center of the image is used as the coordinate point to extract the part that does not contain image information, and calculate the length of each pixel point from the circular area and the angle of each pixel point relative to the center point. Then, the pixel points with image information inside the circle can be mapped to the outside of the circle according to the trigonometric function formula. The capsule endoscopy image after mirroring is shown as follows. Figure 2 (b) as shown.

[0074] It should be understood that the processed image pair includes the mirrored current image and the mirrored next frame image.

[0075] Step S30: performing position matching on each frame of the processed image pair according to a preset SiameseCF network model to obtain a position matching result.

[0076] It should be noted that the preset SiameseCF network model refers to a pre-set model for evaluating image similarity. The SiameseCF network model is based on an end-to-end fully convolutional Siamese (Fully-Convolutional Siamese, Siamese) network algorithm, combined with the correlation operation (Correlation Filter, CF) to obtain the best matching position of the current image on the next frame of the current image. The SiameseCF network regards template matching as a similarity learning process, and the current image in the input image pair and the next frame of the current image are positionally matched to determine the position matching result.

[0077] It is understandable that the role of the preset SiameseCF network model network is not only to determine the position matching result. The network has two parts. The first half is feature extraction, and the second half outputs the optimal offset position of the image block in the current image in the next frame of the current image based on the extracted features.

[0078] It should be understood that, according to the preset SiameseCF network model, the image blocks in each frame of the processed image pair are position matched to obtain the image block position matching result in the image.

[0079] Step S40: determining the image similarity between the current image and the next frame image of the current image according to the position matching result.

[0080] It should be noted that the position matching result refers to the result of matching and outputting the image blocks in the current image in the next frame of the current image. Based on the result, the image similarity between the current image and the next frame of the current image can be judged, that is, the image similarity between the current image and the next frame of the current image is determined based on the position matching result of the image blocks.

[0081] It can be understood that the image similarity is used to screen redundant images, where the redundant images refer to images with high image similarity.

[0082] Step S50: at the end of the traversal, the key frame image in the capsule endoscopy image sequence is determined according to the obtained similarities of all images.

[0083] It should be noted that key frame images refer to the remaining images of the redundant images of the screening image to facilitate the doctor's reading work.

[0084] It is understandable that, at the end of the traversal, redundant images are screened out according to the obtained similarities of all images, and key frame images in the capsule endoscopy image sequence are determined.

[0085] In the specific implementation, the capsule endoscopy image sequence similarity evaluation method based on SiameseCF network uses an integrated descriptor to calculate the similarity between any two capsule endoscopy images. For further explanation, please refer to Figure 3 Schematic diagram of the workflow of capsule endoscopy image sequence similarity based on SiameseCF network. The specific process is as follows: (1) input capsule endoscopy image sequence; (2) take the first image in the sequence as the reference image and perform a mirroring operation; (3) read the next image as the target image and perform a mirroring operation; (4) take the image blocks on the reference image as the template image and the target image as the search image in turn, and send the template image and the search image into the SiameseCF network to match the two capsule endoscopy image blocks and extract features; (5) obtain the integrated descriptor of the two capsule endoscopy images, calculate the similarity by joint probability, if the similarity is less than the threshold, then use the target image as the reference image (the threshold can be set to the empirical value of 0.65, where different thresholds are set according to different applications, and the threshold is not limited to 0.65), if the similarity is greater than the threshold, then the target image is defined as a redundant image to be screened; (6) determine whether the target image is the last image in the sequence according to the sequence number, if not, repeat step 3; (7) screen out the redundant images of the sequence according to the above process (do not delete them from the sequence to ensure the integrity of the image sequence), and output the key frame images of the capsule endoscopy image sequence to facilitate the doctor's reading work.

[0086] This embodiment traverses the images in the capsule endoscope image sequence, and obtains the traversed current image and the next frame of the current image; performs mirror processing on the current image and the next frame of the current image respectively to obtain a processed image pair; performs position matching on the image blocks in each frame of the processed image pair according to the preset SiameseCF network model to obtain the image block position matching result in the image; determines the image similarity between the current image and the next frame of the current image according to the position matching result of the image blocks; at the end of the traversal, determines the key frame image in the capsule endoscope image sequence according to the obtained similarities of all images. Since this embodiment performs position matching on each frame of the image pair after the mirror processing through the preset SiameseCF network model, thereby determining the image similarity between the current image and the next frame of the current image, compared with the prior art that screens and selects films through manual intervention, resulting in a long film reading period and low work efficiency, this embodiment effectively reduces the film reading time and improves work efficiency.

[0087] Reference Figure 4 , Figure 4 This is a flow chart of a second embodiment of a method for determining key frames of a capsule endoscopy image sequence according to the present invention. Figure 1 The first embodiment shown here proposes a second embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention.

[0088] In this embodiment, the step S30 includes:

[0089] Step S301: taking each current image frame in the processed image pair as a reference image, and taking the next image frame of the reference image as a target image.

[0090] Step S302: extracting image blocks of a preset size from the reference image based on a preset interval.

[0091] It should be noted that the preset interval refers to a preset interval for extracting image blocks, and the preset size refers to a preset size for extracting image blocks.

[0092] In the specific implementation, image blocks of fixed size are extracted at equal intervals for each frame of capsule endoscopy image, such as 7×7 pixels, 30×30 pixels, 40×40 pixels, etc. There is no mandatory limit on the image block size. If a frame of capsule endoscopy image is 240×240 pixels, and image blocks of 40×40 pixels are extracted at intervals of 40 pixels, then a capsule endoscopy image will generate 36 image blocks. Figure 5 Schematic diagram of capsule endoscopy images from which 36 image blocks are extracted. The image blocks may be extracted in a non-overlapping manner, and may also be extracted in an overlapping manner, that is, the blocks are overlapped.

[0093] Step S303: inputting the extracted image block and the target image into a preset SiameseCF network model for feature extraction to obtain a feature extraction result.

[0094] It should be noted that the feature extraction result is to input the extracted image block and the target image into the preset SiameseCF network. For further explanation, please refer to Figure 6 Schematic diagram of the SiameseCF network structure. In the model, the ConvNet part extracts the features of the image, and the feature extraction refers to the texture feature extraction of the image block. The texture features of the image block are obtained through the output of the ConvNet part in the SiameseCF network. ConvNet contains 5 convolutional layers, but is not limited to 5 convolutional layers. Convolutional layers can be increased or reduced according to different applications. Its main function is to extract the deep feature information of the input image. The two ConvNets in the SiameseCF network have the same structure and share parameters, so the output of ConvNet can be regarded as the texture features of the input image block.

[0095] It can be understood that ConvNet is a fully convolutional network layer, which is used to extract the features of the input image. For further explanation, please refer to Figure 7 The schematic diagram of the ConvNet network structure is as follows Figure 7 As shown in the figure, except for conv5, each convolution layer is followed by a ReLU nonlinear activation layer, and the pooling layer adopts max-pooling. The network structure in the preset SiameseCF network model can be changed according to the application.

[0096] Step S304: determining the optimal offset positions of all image blocks in the reference image in the target image according to the feature extraction result.

[0097] It should be noted that the best offset position may be the offset position of all image blocks in the reference image with the highest score in the target image.

[0098] In the specific implementation, two capsule endoscopy images to be evaluated are defined as reference image A and target image B. According to the capsule endoscopy image block extraction method, 36 image blocks of reference image A are extracted. An image block in reference image A is used as a template image, and target image B is used as a matching image or search image. These two images are sent to the SiameseCF network to extract image features through the ConvNet part, and the network output score map (i.e., the score of the matching position of the image block in the target image) is obtained through related operations. For further explanation, please refer to Figure 8 Schematic diagram of convolution operation. Figure 8The figure shows a convolution kernel of size m performing related operations on an input image of size n, and the output image size is n-m+1.

[0099] The feature map obtained by the template image through ConvNet is regarded as the convolution kernel. The relevant operation is performed on the feature map obtained by the search image through ConvNet to obtain the best matching position score map of an image block in the reference image A in the target image B. Fig. 9 Schematic diagram of the best offset. The position with the highest score is the best matching position of the image block in image B. (a) is the reference image A, (b) is the score map, and (c) is the target image B.

[0100] Step S305: Determine a position matching result according to the optimal offset position.

[0101] It should be noted that the position matching result of all the image blocks of the reference image in the target image is determined according to the optimal offset positions of all the image blocks in the reference image in the target image.

[0102] In this embodiment, the step S40 includes:

[0103] Step S401: determining an image matching block between the reference image and the target image according to the optimal offset position.

[0104] It should be noted that an image block in the reference image and the target image are input into the SiameseCF network in turn, and the above steps are repeated until the best matching positions of all image blocks in the reference image in the target image are obtained, and the image matching blocks are determined according to the best matching positions.

[0105] In the specific implementation, for further explanation, please refer to Fig.10 Schematic diagram of the matching of two capsule endoscopy image blocks, (d) is the reference image A, (e) is the target image B. An image block in image A and the target image B are sent to the SiameseCF network in turn to obtain the best matching positions of all image blocks in A in the target image B.

[0106] Step S402: determining the position distribution and texture similarity of the image block through an integrated descriptor according to the image matching block, and determining the image similarity between the reference image and the target image according to the position distribution and texture similarity of the image block.

[0107] It should be noted that the image similarity between the reference image and the target image is determined by calculating the probability of the position distribution and texture similarity of the image blocks through the integrated descriptor. The integrated descriptor is a whole that integrates the position and texture information of all image blocks in an image. The descriptor of an image block is a vector that characterizes the texture features of the image block.

[0108] It can be understood that, through the above process, the image blocks in the reference image A and the target image B are sent to the SiameseCF network as the template image and the search image in turn, and the best matching position of the image block of the reference image A in the target image B is obtained, thereby obtaining the image block matching between the two capsule endoscopy images. Based on the above results, the similarity between the two capsule endoscopy images can be evaluated.

[0109] Further, the step S402 includes: calculating the joint probability between the reference image and the target image based on a preset joint probability formula and the image matching block; determining the position distribution similarity and texture similarity of the image block according to the joint probability and the integrated descriptor, and determining the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image block.

[0110] It should be noted that the algorithm uses an integrated descriptor to evaluate the similarity between two capsule endoscopy images. The specific approach is to divide a capsule endoscopy image into 36 40×40 pixel image blocks at equal intervals. Each image block describes the local texture information and records the position information of the image block in the image. The local texture and position information of the image block are combined to comprehensively express the characteristics of a capsule endoscopy image. A capsule endoscopy image is represented by an integrated descriptor of 36 image blocks with relative position information, which can effectively express the texture information of different position areas in the capsule endoscopy image.

[0111] It is understood that for further explanation, reference can be made to Fig.11 Schematic diagram of integrated descriptor similarity evaluation corresponding to some image blocks, as shown in Fig.11 As shown in the figure, 9 image blocks are used to illustrate the similarity evaluation process of the integrated descriptor. Taking the integrated descriptor of the reference image as the benchmark, after the SiameseCF network matches the image blocks, the target image obtains the local texture description image block with the highest similarity to the reference image, as well as the location information of the image block. Fig.11 The target capsule endoscopy image shown in (1) and (2) has a similar scene to the reference image, and the image block position distribution and texture features of the matched integrated descriptor have a high similarity with the reference image. Fig.11 For capsule endoscopy images shown in (3) and (4) that have low similarity with the reference image, the image block position distribution and texture features of the searched integrated descriptor have low similarity with the reference image.

[0112] It should be understood that the preset joint probability formula can be a preset formula for calculating the similarity between the reference image and the target image. If the two capsule endoscopy images whose similarity is to be evaluated are the reference image denoted as A and the target image denoted as B, x is represented as the integrated descriptor of image A, and y is represented as the integrated descriptor of image B, the joint probability P(x, y) of the similarity between image A and image B can be expressed by the conditional probability formula:

[0113] P(x,y)=P(y|x)P(x) Formula (1)

[0114] x is represented as the integrated descriptor in the reference image; y is represented as the integrated descriptor in the target image; then the joint probability between x and y is described as:

[0115]

[0116] in represents the feature descriptor vector of the i-th image patch in x, represents the feature descriptor vector of the i-th image block in y, c x Denoted as the center coordinate of the ensemble descriptor of the reference image, c y Denoted as the center coordinates of the integrated descriptor of the target image, represents the coordinate position of the i-th image block in x, Represents the coordinate position of the i-th image block in y.

[0117] If the integrated descriptor of the reference image and the integrated descriptor of the target image are independent of each other, then according to formula (1) and conditional independence, the joint probability P(x, y) is decomposed into:

[0118]

[0119] Among them, the conditional probability in formula (3) It is represented as the feature descriptor vector of the i-th image block in the reference image integrated descriptor x in the image pair The corresponding target image integrated descriptor y is the i-th image block feature descriptor vector degree of similarity;

[0120] The similarity between image matching patch descriptors is modeled by a Gaussian distribution:

[0121]

[0122] Formula (4) represents the similarity between the image block feature descriptor vector of the reference image and the image block feature descriptor vector of the target image, α1 is a constant, S D is the constant covariance matrix

[0123] The similarity between the feature descriptor vectors of the image matching blocks is determined according to the above formulas (1)-(4). The similarity between the feature descriptor vectors can determine the texture similarity of the image matching blocks.

[0124] In order to obtain image similarity, it is necessary to determine the position distribution similarity and texture similarity of the image blocks based on the joint probability and the integrated descriptor, and determine the image similarity between the reference image and the target image based on the position distribution similarity and texture similarity of the image blocks.

[0125] It should be noted that the conditional probability Denoted as the relative position of the i-th image block in the reference image integrated descriptor x The corresponding relative position of the i-th image block in the target image is Since the relative position relationship between image blocks cannot be analyzed by formula, for the convenience of calculation, if the positions of each image block in the reference image and the positions of each image block in the target image are independent of each other, Gaussian distribution is used to model the similarity of matching block positions:

[0126]

[0127] Formula (5) represents the position similarity between the image block of the reference image and the image block of the target image, α2 is a constant, S L is the constant covariance matrix.

[0128] The conditional probability P(c y |c x ), They represent the similarity of the central image blocks of the set descriptors x and y and the offset of each image block in the set descriptor y relative to the central coordinates, which can be obtained using formulas (6)-(7)

[0129]

[0130]

[0131] Formula (6) indicates that if the similarity of the central image blocks of the integrated descriptors x and y is higher than the threshold τ, the value is 1, otherwise it is 0. Formula (7) indicates the degree of deviation of each image block in the integrated descriptor y relative to the central coordinates, and the value is 0 if it exceeds the image range, otherwise it is 1.

[0132] According to the above formula and the decomposition of the joint probability P(x, y) in formula (7), in this experiment, the log-likelihood function is used to represent the similarity between the integrated descriptors x and y, and the similarity evaluation of images A and B is obtained by normalizing them using formula (10).

[0133]

[0134] max=-logP(x0,y0) formula (9)

[0135]

[0136] The similarity between the integrated descriptors x and y can be obtained from formula (8). In formula (9), x0 and y0 are the integrated descriptors obtained after matching the image blocks of the pure white image and the pure black image, respectively. The calculated max is the theoretical upper limit of formula (8). Formula (10) normalizes the similarity value obtained by formula (9) to the range of [0,1]. The higher the value, the more similar the target image B is to the reference image A.

[0137] In order to determine the image similarity, the steps of determining the position distribution similarity and texture similarity of the image blocks according to the joint probability and the integrated descriptor, and determining the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image blocks, include: if the feature descriptor vectors of each image block of the reference image and the feature descriptor vectors of each image block of the target image are independent of each other, calculating the similarity between the feature descriptor vectors corresponding to the image matching blocks by using the integrated descriptor and the joint probability; if the positions of each image block in the reference image and the positions of each image block in the target image are independent of each other, calculating the position similarity of the image matching blocks by using the integrated descriptor and the joint probability; and determining the image similarity between the reference image and the target image according to the similarity between the feature descriptor vectors, the position similarity and a preset similarity threshold.

[0138] In this embodiment, images in a capsule endoscopy image sequence are traversed, and the traversed current image and the next frame image of the current image are obtained; the current image and the next frame image of the current image are mirrored respectively to obtain a processed image pair; each current frame image in the processed image pair is used as a reference image, and the next frame image of the reference image is used as a target image; image blocks of a preset size are extracted from the reference image based on a preset interval; the extracted image blocks and the target image are input into a preset SiameseCF network model for feature extraction to obtain feature extraction results; the optimal offset positions of all image blocks in the reference image in the target image are determined according to the feature extraction results; the position matching results are determined according to the optimal offset positions, and the image matching blocks between the reference image and the target image are determined according to the optimal offset positions; the image matching blocks determine the position distribution and texture similarity of the image blocks through an integrated descriptor, and determine the image similarity between the reference image and the target image according to the position distribution and texture similarity of the image blocks. Since the present embodiment uses a preset SiameseCF network model to perform position matching on each frame of the mirrored image pair, thereby determining the image similarity between the current image and the next frame of the current image, compared with the prior art that uses manual intervention to screen and select films, resulting in a long film reading period and low work efficiency, the present embodiment effectively reduces the film reading time and improves work efficiency.

[0139] Reference Fig.12 , Fig.12 The flowchart of the third embodiment of the method for determining key frames of capsule endoscopy image sequence of the present invention is based on the above Figure 1 The first embodiment shown here is a third embodiment of the method for determining key frames of a capsule endoscopy image sequence of the present invention.

[0140] In this embodiment, before step S10, the method further includes:

[0141] Step S01: performing a circular mirroring operation on each frame of the historical capsule endoscopy image sequence to obtain a mirrored capsule endoscopy image sequence.

[0142] Step S02: traversing the mirrored capsule endoscopy image sequence, taking two adjacent similar images traversed as a training sample pair, and generating a sample set according to the training sample pair.

[0143] In the specific implementation, the algorithm takes two adjacent similar images in the capsule endoscopy image sequence as a training sample pair, takes the central image block of the previous image as the template image x, and the latter image as the search image y to generate training data, and the label value takes the central image block of the search image. For further explanation, please refer to Fig.13Schematic diagram of training sample pairs. The image block enclosed by the box in the search image is the label value of the template image. There can be an appropriate degree of offset when taking the label value to avoid overfitting during the training process. The sizes of the template image and the search image are 40 pixels and 240 pixels respectively. Directly sending the template image to the network will result in a feature map size that is too small and not easy for subsequent related operations. Therefore, appropriate scaling transformation is performed to convert the template image size to 120 pixels to facilitate subsequent experiments. The specific parameters are shown in Table 1.

[0144] Input template image x and search image y, determine the position of the template image in the search image by judging the degree of similarity between x and y, regard the feature map of x obtained by ConvNet as the convolution kernel, perform relevant operations on the feature map of y obtained by ConvNet, and output the score map of the best matching position.

[0145] The ConvNet network architecture is shown in Table 1, which gives the specific parameters of the convolution layer and the pooling layer, as well as the size of the input reference image and the target image after passing through each layer of the network (for example only, the input image size can be adjusted according to the image size of the sample set).

[0146] Table 1 ConvNet network architecture

[0147]

[0148] Step S03: training an initial SiameseCF network model according to the sample set to obtain a preset SiameseCF network model.

[0149] It should be noted that, the initial SiameseCF network model is trained according to the sample set to obtain a preset SiameseCF network model for matching the offset of image blocks between capsule endoscopy images.

[0150] Furthermore, the step S03 includes: using the central image block of each frame of the current image in the sample pair as a template image, and the next frame image of the current image as a search image; inputting the template image and the search image into the original SiameseCF network model to obtain a score map of the template image on the search image; calculating the loss of the score map by a preset label value, and using a preset stochastic gradient descent method and the loss to train the network parameters of the original SiameseCF network model to obtain a preset SiameseCF network model.

[0151] It should be noted that the preset label value may refer to a preset loss function for calculating the score map. The loss function is used to measure the quality of the network. The label value and the score map output by the network are calculated by the formula to obtain the loss, and the parameters of the network are automatically optimized by the stochastic gradient descent method. The optimal preset SiameseCF network model is obtained by repeated training.

[0152] In the specific implementation, the template image and the search image in the sample pair of the above dataset are sent to the SiameseCF network in sequence. The network outputs the score map of the template image on the search image. The loss of the score map is calculated by the label value, and the network parameters are trained by the stochastic gradient descent method.

[0153] The loss for each coordinate position score in the network output score map is defined as follows:

[0154] l(z,v)=log(1+exp(-zv)) Formula (11)

[0155] In formula (11), v represents the score value of each position in the score map, z∈(+1, -1) is the label value corresponding to this coordinate position, and the loss of each coordinate score is defined by the logarithmic (log) loss function. The label of each position in the score map is defined as +1 within the radius R of the center coordinate, and -1 otherwise.

[0156]

[0157] In formula (12), c is the center coordinate of the score map, u is the coordinate of each point in the score map, and k is the compression ratio of the search image to the score map size. In the SiameseCF network, due to the use of full convolution, compared with the traditional twin network, the target can be found in a larger image range, and the output is a score map, not just a value. Therefore, during the training process, the overall loss of the score map is defined as the average of all position losses, which is defined as follows:

[0158]

[0159] In formula (13), D is the area of ​​the score map.

[0160] For the parameters θ of the SiameseCF network, Stochastic Gradient Descent (SGD) is used for training

[0161]

[0162] Through the above data set, loss function and SGD, the SiameseCF network model can be obtained through training.

[0163] This embodiment performs a circular mirroring operation on each frame of the historical capsule endoscopy image sequence to obtain a mirrored capsule endoscopy image sequence; traverses the mirrored capsule endoscopy image sequence, and uses the two adjacent similar images traversed as a training sample pair, and generates a sample set based on the training sample pair; trains the initial SiameseCF network model based on the sample set to obtain a preset SiameseCF network model. Traverse the images in the capsule endoscopy image sequence, and obtain the traversed current image and the next frame of the current image; mirror the current image and the next frame of the current image respectively to obtain a processed image pair; perform position matching on the image blocks in each frame of the processed image pair based on the preset SiameseCF network model to obtain the image block position matching result in the image; determine the image similarity between the current image and the next frame of the current image based on the image block position matching result; at the end of the traversal, determine the key frame image in the capsule endoscopy image sequence based on the obtained similarities of all images. Since the present embodiment uses a preset SiameseCF network model to perform position matching on each frame of the mirrored image pair, thereby determining the image similarity between the current image and the next frame of the current image, compared with the prior art that uses manual intervention to screen and select films, resulting in a long film reading period and low work efficiency, the present embodiment effectively reduces the film reading time and improves work efficiency.

[0164] Reference Fig.14 , Fig.14 It is a structural block diagram of the first embodiment of the capsule endoscopy image sequence key frame determination system of the present invention.

[0165] like Fig.14 As shown, the capsule endoscopy image sequence key frame determination system proposed in the embodiment of the present invention may include: an image traversal module 10, a mirror processing module 20, a feature extraction module 30, a similarity determination module 40, and a key frame image determination module 50;

[0166] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the system for determining key frames of capsule endoscopy image sequences, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0167] The embodiment of the present invention provides a capsule endoscopy image sequence key frame determination system, referring to Fig.14 , Fig.14 The structure diagram of the capsule endoscopy image sequence key frame determination system of the present invention is shown.

[0168] The image traversal module 10 in this embodiment is used to traverse the images in the capsule endoscopy image sequence and obtain the traversed current image and the next frame image of the current image;

[0169] A mirror processing module 20, used to perform mirror processing on the current image and the next frame image of the current image respectively to obtain a processed image pair;

[0170] A feature extraction module 30 is used to perform position matching on each frame of the processed image pair according to a preset SiameseCF network model to obtain a position matching result;

[0171] A similarity determination module 40 is used to determine the image similarity between the current image and the next frame image of the current image according to the position matching result;

[0172] The key frame image determination module 50 is used to determine the key frame image in the capsule endoscopy image sequence according to the obtained similarities of all images at the end of the traversal.

[0173] This embodiment traverses the images in the capsule endoscope image sequence, and obtains the traversed current image and the next frame of the current image; performs mirror processing on the current image and the next frame of the current image respectively to obtain a processed image pair; performs position matching on the image blocks in each frame of the processed image pair according to the preset SiameseCF network model to obtain the image block position matching result in the image; determines the image similarity between the current image and the next frame of the current image according to the position matching result of the image blocks; at the end of the traversal, determines the key frame image in the capsule endoscope image sequence according to the obtained similarities of all images. Since this embodiment performs position matching on each frame of the image pair after the mirror processing through the preset SiameseCF network model, thereby determining the image similarity between the current image and the next frame of the current image, compared with the prior art that screens and selects films through manual intervention, resulting in a long film reading period and low work efficiency, this embodiment effectively reduces the film reading time and improves work efficiency.

[0174] Furthermore, the feature extraction module 30 is also used to use each current image frame in the processed image pair as a reference image and the next frame image of the reference image as a target image; extract image blocks of a preset size from the reference image based on a preset interval; input the extracted image blocks and the target image into a preset SiameseCF network model for feature extraction to obtain feature extraction results; determine the optimal offset positions of all image blocks in the reference image in the target image based on the feature extraction results; and determine the position matching results based on the optimal offset positions.

[0175] Furthermore, the similarity determination module 40 is also used to determine the image matching block between the reference image and the target image according to the optimal offset position; determine the position distribution similarity and texture similarity of the image block through an integrated descriptor according to the image matching block, and determine the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image block.

[0176] Furthermore, the similarity determination module 40 is also used to calculate the joint probability between the reference image and the target image based on a preset joint probability formula and the image matching block; determine the position distribution similarity and texture similarity of the image block according to the joint probability and the integrated descriptor, and determine the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image block.

[0177] Furthermore, the similarity determination module 40 is also used to, if the feature descriptor vectors of each image block of the reference image and the feature descriptor vectors of each image block of the target image are independent of each other, calculate the similarity between the feature descriptor vectors corresponding to the image matching blocks by integrating the descriptor and the joint probability; if the positions of each image block in the reference image and the positions of each image block in the target image are independent of each other, calculate the position similarity of the image matching blocks by integrating the descriptor and the joint probability; determine the image similarity between the reference image and the target image according to the similarity between the feature descriptor vectors, the position similarity and a preset similarity threshold.

[0178] Furthermore, the capsule endoscopy image sequence key frame determination system also includes a model construction module, which is used to perform a circular mirror operation on each frame of the historical capsule endoscopy image sequence to obtain a mirrored capsule endoscopy image sequence; traverse the mirrored capsule endoscopy image sequence, and use the traversed two adjacent similar images as a training sample pair, and generate a sample set based on the training sample pair; train an initial SiameseCF network model based on the sample set to obtain a preset SiameseCF network model.

[0179] Furthermore, the model building module is also used to use the central image block of each frame of the current image in the sample pair as a template image, and the next frame image of the current image as a search image; input the template image and the search image into the original SiameseCF network model to obtain a score map of the template image on the search image; calculate the loss of the score map through a preset label value, and use a preset stochastic gradient descent method and the loss to train the network parameters of the original SiameseCF network model to obtain a preset SiameseCF network model.

[0180] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0181] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

Claims

1. A method for determining key frames of capsule endoscopy image sequence, characterized in that: The method for determining key frames of capsule endoscopy image sequence comprises the following steps: Traversing images in a capsule endoscopy image sequence, and acquiring a current image traversed and a next frame image of the current image; Performing mirror processing on the current image and the next frame of the current image respectively to obtain a processed image pair; Performing position matching on each frame of the processed image pair according to a preset SiameseCF network model to obtain a position matching result; Determining the image similarity between the current image and the next frame image of the current image according to the position matching result; At the end of the traversal, determining a key frame image in the capsule endoscopy image sequence according to the obtained similarities of all images; The step of performing position matching on each frame of the processed image pair according to the preset SiameseCF network model to obtain a position matching result comprises: Using each current image in the processed image pair as a reference image and the next image of the reference image as a target image; extracting an image block of a preset size from the reference image based on a preset interval; Inputting the extracted image block and the target image into a preset SiameseCF network model for feature extraction to obtain a feature extraction result; Determine the optimal offset positions of all image blocks in the reference image in the target image according to the feature extraction result; Determine a position matching result according to the optimal offset position; The step of determining the image similarity between the current image and the next frame image of the current image according to the position matching result comprises: determining an image matching block between the reference image and the target image according to the optimal offset position; Calculating a joint probability between the reference image and the target image based on a preset joint probability formula and the image matching block; The position distribution similarity and texture similarity of the image blocks are determined according to the joint probability and the integrated descriptor, and the image similarity between the reference image and the target image is determined according to the position distribution similarity and texture similarity of the image blocks.

2. The method for determining key frames of capsule endoscopy image sequence according to claim 1, characterized in that: The step of determining the position distribution similarity and texture similarity of the image blocks according to the joint probability and the integrated descriptor, and determining the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image blocks comprises: If the feature descriptor vectors of each image block of the reference image and the feature descriptor vectors of each image block of the target image are independent of each other, the similarity between the feature descriptor vectors corresponding to the image matching blocks is calculated by integrating the descriptor and the joint probability; If the positions of the image blocks in the reference image and the positions of the image blocks in the target image are independent of each other, calculating the position similarity of the image matching blocks by integrating the descriptor and the joint probability; The image similarity between the reference image and the target image is determined according to the similarity between the feature descriptor vectors, the position similarity and a preset similarity threshold.

3. The method for determining key frames of capsule endoscopy image sequence according to any one of claims 1 or 2, characterized in that: Before the step of traversing the images in the capsule endoscopy image sequence and acquiring the traversed current image and the next frame image of the current image, the method further includes: Performing a circular mirror operation on each frame of the historical capsule endoscopy image sequence to obtain a mirrored capsule endoscopy image sequence; Traversing the mirrored capsule endoscopy image sequence, taking two adjacent similar images traversed as a training sample pair, and generating a sample set according to the training sample pair; An initial SiameseCF network model is trained according to the sample set to obtain a preset SiameseCF network model.

4. The method for determining key frames of capsule endoscopy image sequence according to claim 3, characterized in that: The step of training the original SiameseCF network model according to the sample set to obtain the preset SiameseCF network model includes: Using the central image block of each current image in the sample pair as a template image, and the next frame of the current image as a search image; Inputting the template image and the search image into the original SiameseCF network model to obtain a score map of the template image on the search image; The loss of the score graph is calculated by a preset label value, and the network parameters of the original SiameseCF network model are trained using a preset stochastic gradient descent method and the loss to obtain a preset SiameseCF network model.

5. A capsule endoscopy image sequence key frame determination system, characterized in that: The capsule endoscopy image sequence key frame determination system comprises: An image traversal module is used to traverse images in a capsule endoscopy image sequence and obtain a current image traversed and a next frame image of the current image; A mirror processing module, used to perform mirror processing on the current image and the next frame image of the current image respectively to obtain a processed image pair; A feature extraction module, used to use each current image in the processed image pair as a reference image and the next image of the reference image as a target image; The feature extraction module is further used to extract image blocks of a preset size from the reference image based on a preset interval; The feature extraction module is further used to input the extracted image block and the target image into a preset SiameseCF network model for feature extraction to obtain a feature extraction result; The feature extraction module is further used to determine the optimal offset positions of all image blocks in the reference image in the target image according to the feature extraction result; The feature extraction module is further used to determine a position matching result according to the optimal offset position; a similarity determination module, configured to determine an image matching block between the reference image and the target image according to the optimal offset position; The similarity determination module is further used to calculate the joint probability between the reference image and the target image based on a preset joint probability formula and the image matching block; determine the position distribution similarity and texture similarity of the image blocks according to the joint probability and the integrated descriptor, and determine the image similarity between the reference image and the target image according to the position distribution similarity and texture similarity of the image blocks; The key frame image determination module is used to determine the key frame image in the capsule endoscopy image sequence according to the obtained similarities of all images at the end of the traversal.

Images