Method and device for analyzing small intestine power system based on endoscopic image sequence

Abstract

The invention provides a method and device for analyzing a small intestine power system based on an endoscope image sequence, and belongs to the field of computer analysis of medical image.The method comprises the steps that an image sequence in a small intestine is collected through a wireless capsule endoscope, and an image in the image sequence is divided into a plurality of image blocks at equal intervals; performing feature extraction and matching on each image block by using a Siamese CF network, calculating the similarity between adjacent image blocks, and performing segmentation to obtain a plurality of scene segments; performing motion analysis on adjacent images in the scene segment by adopting a double-reference deformation field evaluation model based on unsupervised contrast learning to obtain a motion analysis result; and calculating the motion displacement and the displacement direction between the images according to the motion analysis result, identifying and classifying the motion mode of the small intestine, and extracting the key frame sequence of the motion of the small intestine according to the motion analysis result. According to the invention, non-invasive evaluation of the small intestine power system is realized, the motion mode of the small intestine is accurately identified, and the key frame sequence is extracted.

Description

Technical Field

[0001] The present invention relates to the technical field of computer analysis of medical images, and in particular to a method and device for analyzing a small intestinal power system based on an endoscopic image sequence. Background Art

[0002] Wireless Capsule Endoscopy (WCE) is a new type of non-invasive gastrointestinal examination tool, referred to as capsule endoscopy or capsule endoscopy. Clinically, after the patient swallows the capsule endoscopy into the human body, it moves forward under its own gravity and the peristalsis of the gastrointestinal tract, and the capsule endoscopy imaging system automatically captures the digestive tract scene at a speed of two frames per second. The captured image data is transmitted to the external receiving terminal in a wireless transmission manner. During the average 7-8 hours of digestive tract examination, tens of thousands of color digestive tract scene images can be collected.

[0003] Normal peristalsis of the small intestine is the basis for nutrients to fully contact the intestinal wall and be effectively absorbed. The peristalsis of the small intestine is also the direct power source for the movement of the contents of the digestive tract. Motility disorders can lead to poor nutrient absorption and cause health problems such as nutritional deficiencies. At the same time, small intestinal peristalsis helps mix digestive juices and chyme and promotes the action of digestive enzymes. If the peristaltic function is abnormal, it may lead to insufficient mixing of chyme and digestive juices, affecting digestion. Insufficient small intestinal motility may also lead to bacterial overgrowth syndrome, causing adverse symptoms such as abdominal distension and diarrhea. Evaluation of the small intestinal motility system is a key method to understand and diagnose small intestinal motility dysfunction. The motility function of the small intestine has an important impact on the function of the digestive system. Evaluation of the small intestinal motility system plays an important role in the diagnosis and management of digestive system dysfunction.

[0004] In the prior art, small intestinal manometry is the most widely accepted and reliable method for evaluating small intestinal motility. However, manometry is an invasive measurement method that determines the motility of the small intestine by measuring the pressure at certain points in the intestine. Although a large number of digestive tract images can be collected by WCE technology in the prior art, there is a lack of effective methods for extracting and analyzing motion information in WCE image sequences.

[0005] Therefore, finding a method that can both non-invasively evaluate small intestinal motility and improve the efficiency and accuracy of small intestinal motility evaluation is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] In view of this, the present invention proposes a method and device for analyzing the small intestinal motility system based on endoscopic image sequences, which can non-invasively evaluate the small intestinal motility system, accurately identify the movement pattern of the small intestine and extract key frame sequences.

[0007] The technical solution of the present invention is achieved in this way:

[0008] In a first aspect, the present invention provides a method for analyzing the small intestinal motility system based on an endoscopic image sequence, comprising the following steps:

[0009] S1, collecting an image sequence of the small intestine through a wireless capsule endoscope, and dividing the images in the image sequence into a plurality of image blocks at equal intervals;

[0010] S2, use SiameseCF network to extract and match features of each image block, calculate the similarity between adjacent image blocks, segment the image sequence according to the similarity threshold, and obtain multiple scene segments;

[0011] S3, using a dual-reference deformation field evaluation model based on unsupervised contrastive learning to perform motion analysis on adjacent images within the scene segment to obtain motion analysis results;

[0012] S4. Calculate the motion displacement and displacement direction between images according to the motion analysis results, identify and classify the motion pattern of the small intestine based on the motion displacement and displacement direction, and extract the key frame sequence of the small intestine movement according to the motion analysis results.

[0013] On the basis of the above technical solution, preferably, the dual-reference deformation field evaluation model based on unsupervised contrastive learning includes:

[0014] A feature extraction module configured to extract texture features of any two adjacent image blocks, a first texture feature and a second texture feature;

[0015] A feature fusion module configured to fuse the first texture feature and the second texture feature to obtain a fused feature;

[0016] A spatial transformation network module is configured to deform the floating image toward the reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks are reference images and floating images to each other;

[0017] An optimization module is configured to optimize parameters of a dual-reference deformation field evaluation model for unsupervised contrastive learning through dual-reference constraints and a loss function.

[0018] Based on the above technical solution, preferably, the formula of the loss function is:

[0019]

[0020] Where L(R1, R2, φ) represents the loss function, R1 represents the first image block, R2 represents the second image block adjacent to the first image block, φ represents the deformation field, R represents any image in R1 or R2, and w R represents the weight of each deformation reference item, represents the similarity loss term, λ represents the regularization term coefficient, represents the regularization term of the deformation field, p represents the pixel position in the deformation field, Ω represents the domain of the deformation field, represents the local differential of the deformation field at position p, and ||·||2 represents the L2 norm.

[0021] On the basis of the above technical solution, preferably, the motion analysis result includes the dual-channel displacement of the image block and the HSV color image after the displacement vector transformation, and step S4 specifically includes:

[0022] Extract dual-channel displacement information and calculate average displacement and direction statistics;

[0023] Use deep convolutional neural networks to extract deep features, analyze the movement characteristics of the small intestine, and classify the movement patterns into contraction, relaxation, and others;

[0024] Filter the output motion pattern sequence and draw the contraction and relaxation curve;

[0025] When the average pixel displacement is large and the displacement direction is multi-directionally distributed, and / or the cumulative displacement in the same scene segment exceeds a preset threshold, the corresponding image is extracted as a key frame.

[0026] On the basis of the above technical solution, preferably, completing one contraction and one relaxation is recorded as one cycle, and the movement state of the small intestine is analyzed by calculating the small intestine contraction frequency, wherein the calculation formula of the small intestine contraction frequency is:

[0027]

[0028] Among them, average-number represents the frequency of small intestine contraction.

[0029] More preferably, step S2 specifically includes:

[0030] S21, taking two adjacent similar images in the image sequence as training sample pairs, constructing a sample set to train the SiameseCF network model;

[0031] S22, taking the first image in the image sequence as a reference image and performing a mirroring operation, reading in the next image as a target image and performing a mirroring operation;

[0032] S23, inputting the image blocks of the reference image and the image blocks of the target image as the template image and the search image into the SiameseCF network respectively, and performing matching and feature extraction between the image blocks;

[0033] S24, calculating the similarity between the template image and the search image by using the Gaussian formula, and when the similarity is less than the similarity threshold, taking the target image as the beginning of a new scene segment and setting it as a new reference image; when the similarity is greater than or equal to the similarity threshold, saving the target image in the current scene segment;

[0034] S25. Repeat steps S21-S24 until the entire image sequence is processed to obtain multiple scene segments.

[0035] Based on the above technical solution, preferably, the similarity is evaluated using an integrated descriptor, specifically including:

[0036] Taking the integrated descriptor of the reference image as the benchmark, after matching the image blocks through the SiameseCF network, the target image obtains the local texture description image block with the highest similarity to the reference image, as well as the location information of the image block.

[0037] In a second aspect, the present invention provides a device for analyzing the small intestinal motility system based on an endoscopic image sequence, using the method described above, comprising:

[0038] A feature extraction module configured to extract texture features of any two adjacent image blocks, a first texture feature and a second texture feature;

[0039] A feature fusion module configured to fuse the first texture feature and the second texture feature to obtain a fused feature;

[0040] A spatial transformation network module is configured to deform the floating image toward the reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks are reference images and floating images to each other;

[0041] An optimization module is configured to optimize parameters of a dual-reference deformation field evaluation model for unsupervised contrastive learning through dual-reference constraints and a loss function.

[0042] In a third aspect, the present invention provides an electronic device, characterized in that it comprises: at least one processor, at least one memory, a communication interface and a bus, wherein:

[0043] The processor, memory, and communication interface communicate with each other via the bus;

[0044] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method described above.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method described above.

[0046] Compared with the prior art, the method for analyzing the small intestinal dynamic system based on endoscopic image sequences of the present invention has the following beneficial effects:

[0047] (1) By performing scene segmentation and displacement field analysis on capsule endoscopy image sequences, a non-invasive assessment of the small intestine's dynamic system is achieved. The small intestine's motion pattern can be accurately identified and key frame sequences can be extracted, providing an objective and reliable basis for clinical diagnosis.

[0048] (2) By adopting a dual-reference deformation evaluation method, the two images to be registered can be used as references to each other and jointly affect the adjustment of network parameters, thereby increasing the constraints of the dual references, improving the accuracy of deformation registration, and making the displacement field evaluation results more accurate and reliable;

[0049] (3) By using an integrated descriptor to evaluate the similarity between two capsule endoscopy images, the images were divided into 36 image blocks and combined with position information, which effectively expressed the texture information of different position areas in the capsule endoscopy images, improved the accuracy of scene segmentation, and provided a reliable basis for subsequent motion analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a flow chart of the method for analyzing the small intestine dynamic system based on endoscopic image sequence of the present invention;

[0052] Figure 2 It is a flowchart of the method for analyzing the small intestine power system based on endoscopic image sequence of the present invention;

[0053] Figure 3 It is a flow chart of scene segmentation based on SiameseCF network capsule endoscopy image sequence similarity of the method for analyzing the small intestine dynamic system based on the endoscopy image sequence of the present invention;

[0054] Figure 4 Schematic diagram of image block distribution of images with different similarities in the method for analyzing the small intestinal dynamic system based on endoscopic image sequence of the present invention; wherein (a) (b) are scenes in which the target capsule endoscopy image and the reference image have similarities, and (c) (d) are capsule endoscopy images with lower similarity to the reference image;

[0055] Figure 5This is a SiameseCF network structure diagram of the method for analyzing the small intestinal dynamic system based on endoscopic image sequences of the present invention;

[0056] Figure 6 This is a SiameseCF network ConvNet structure diagram of the method for analyzing the small intestine power system based on endoscopic image sequences of the present invention;

[0057] Figure 7 It is a schematic diagram of the method for analyzing the small intestine dynamic system based on endoscopic image sequences according to the present invention based on unsupervised dual reference deformation field evaluation;

[0058] Figure 8 A flow chart of the method for analyzing the small intestine dynamic system based on endoscopic image sequences according to the present invention based on unsupervised dual reference deformation field evaluation;

[0059] Fig. 9 A schematic diagram of a small intestinal motility analysis network of the method for analyzing the small intestinal motility system based on endoscopic image sequences of the present invention;

[0060] Fig.10 It is a flow chart of motion analysis and key frame extraction of the method for analyzing the small intestine dynamic system based on endoscopic image sequence of the present invention;

[0061] Fig.11 Schematic diagram of the modified VGG network structure of the method for analyzing the small intestinal dynamic system based on endoscopic image sequences of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] like Figure 1 and Figure 2 As shown, the present invention provides a method for analyzing the small intestinal motility system based on endoscopic image sequences, comprising the following steps:

[0064] S1. Collect an image sequence in the small intestine through a wireless capsule endoscope, and divide the images in the image sequence into multiple image blocks at equal intervals.

[0065] In one embodiment of the present application, a capsule endoscopy image is equally divided into 36 40×40 pixel image blocks, each image block describes local texture information, and simultaneously records the position information of the image block in the image. The local texture and position information of the image block are combined to comprehensively express the characteristics of a capsule endoscopy image.

[0066] S2. Use the SiameseCF network to extract and match features of each image block, calculate the similarity between adjacent image blocks, and segment the image sequence according to the similarity threshold to obtain multiple scene segments.

[0067] like Figure 3 and Figure 4 As shown, specifically, step S2 specifically includes:

[0068] S21, taking two adjacent similar images in the image sequence as training sample pairs, constructing a sample set to train the SiameseCF network model;

[0069] S22, taking the first image in the image sequence as a reference image and performing a mirroring operation, reading in the next image as a target image and performing a mirroring operation;

[0070] S23, inputting the image blocks of the reference image and the image blocks of the target image as the template image and the search image into the SiameseCF network respectively, and performing matching and feature extraction between the image blocks;

[0071] S24, calculating the similarity between the template image and the search image by using the Gaussian formula, and when the similarity is less than the similarity threshold, taking the target image as the beginning of a new scene segment and setting it as a new reference image; when the similarity is greater than or equal to the similarity threshold, saving the target image in the current scene segment;

[0072] S25. Repeat steps S21-S24 until the entire image sequence is processed to obtain multiple scene segments.

[0073] In a preferred embodiment of the present application, the similarity threshold is set to 0.65.

[0074] like Figure 3 As shown, it can be understood that the SiameseCF network regards target tracking as a similarity learning process. It inputs the template image x and the search image y, determines the position of the template image in the search image by judging the similarity between x and y, regards the feature map obtained by the ConvNet of the template image as the convolution kernel, performs relevant operations on the feature map obtained by the ConvNet of the search image, and outputs the score map of the best matching position.

[0075] Among them, ConvNet in SiameseCF network is a fully convolutional network layer, which is used to extract the features of the input image. The network structure is as follows: Figure 5 and Figure 6As shown in the figure, except for conv5, each convolution layer is followed by a ReLU nonlinear activation layer, and the pooling layer uses max-pooling. The network architecture is shown in Table 1, which gives the specific parameters of the convolution layer and the pooling layer, as well as the size of the input template image x and the search image y after passing through each layer of the network (for example only, the input image size can be adjusted according to the sample set image size).

[0076] Table 1

[0077]

[0078] Furthermore, the similarity is evaluated using an integrated descriptor, specifically including:

[0079] Taking the integrated descriptor of the reference image as the benchmark, after matching the image blocks through the SiameseCF network, the target image obtains the local texture description image block with the highest similarity to the reference image, as well as the location information of the image block.

[0080] In one embodiment of the present application, a capsule endoscopy image is represented by an integrated descriptor of 36 image blocks with relative position information, which can effectively express the texture information of different position areas in the capsule endoscopy image.

[0081] This application uses an integrated descriptor to evaluate the similarity between two capsule endoscopy images, divides the images into 36 image blocks and combines the position information, effectively expressing the texture information of different position areas in the capsule endoscopy image, improving the accuracy of scene segmentation, and providing a reliable basis for subsequent motion analysis.

[0082] like Figure 4 As shown, in one embodiment of the present application, 10 image blocks are used to illustrate the similarity evaluation process of the integrated descriptor. Figure 4 The target capsule endoscopy image shown in (a) and (b) has a similar scene to the reference image, and the image block position distribution and texture features of the matched integrated descriptor have a high similarity with the reference image. Figure 4 For capsule endoscopy images shown in (c) and (d) that have low similarity with the reference image, the image block position distribution and texture features of the searched integrated descriptor have low similarity with the reference image.

[0083] Assume that the two capsule endoscopy images to be evaluated are reference image A and target image B. The integrated descriptors in images A and B are represented by x and y respectively. The joint probability P(x, y) of the similarity between images A and B can be expressed as:

[0084] P(x,y)=P(y|x)P(x) (1)

[0085] Among them, P(x,y) represents the joint probability distribution of reference image A and target image B, p(y|x) represents the probability of occurrence of integrated descriptor y of target image B when the integrated descriptor x of reference image A is known, and P(x) represents the marginal probability distribution of integrated descriptor x of reference image A.

[0086] The value of P(x,y) is between 0 and 1. The closer the value is to 1, the more similar the integrated descriptors x and y are.

[0087] make represents the descriptor vector of the i-th image block in x, represents the descriptor vector of the i-th image block in y, c x and c y represents the center coordinates of the integrated descriptor of images A and B, represents the coordinate position of the i-th image block in x, represents the coordinate position of the i-th image block in y. Then the joint probability between x and y can be described as:

[0088]

[0089] in, represents the descriptor vector of the first image block in x, Indicates the coordinate position of the first image block in x, Represents the descriptor vector of the first image block in y.

[0090] Assuming that the two integrated descriptors x and y are relatively independent, according to formula (1) and conditional independence, the joint probability P(x, y) can be decomposed into:

[0091]

[0092] Where n represents the total number of image blocks, represents the conditional probability of the i-th image block descriptor in the target image B given the i-th image block descriptor in the reference image A, represents the conditional probability of the i-th image block position in the target image B under the condition that the i-th image block position in the reference image A and the center coordinates of the two image integrated descriptors are known, P(c y |c x ) represents the conditional probability of the center coordinates of the integrated descriptor of the target image B appearing under the condition that the center coordinates of the integrated descriptor of the reference image A are given.

[0093] It is assumed that the ensemble descriptor vectors are independent of each other, and a Gaussian distribution is used to model the similarity between matching patch descriptors:

[0094]

[0095] Where α1 is a constant, S D is the constant covariance matrix, which determines the allowed deviation between the descriptor vectors.

[0096] Since the mutual position relationship between image blocks cannot be analyzed by formula, for the convenience of calculation, it is assumed that the position of each image block in the image is relatively independent, and Gaussian distribution is used to model the similarity of matching block positions:

[0097]

[0098] Where α2 is a constant, S L is a constant covariance matrix that determines the allowed deviation between the relative positions of image patches.

[0099] The conditional probability P(c y |c x ), They represent the similarity of the central image blocks of the set descriptors x and y and the offset of each image block in the set descriptor y relative to the central coordinates, which can be obtained using formulas (6)-(7)

[0100]

[0101] Formula (6) indicates that if the similarity of the central image blocks of the set descriptors x and y is higher than the threshold τ, the value is 1, otherwise it is 0. Formula (7) indicates the degree of deviation of each image block in the set descriptor y relative to the central coordinates, and the value is 0 if it exceeds the image range, otherwise it is 1.

[0102] According to the above formula and the decomposition of the joint probability P(x, y) in formula (7), in this experiment, the log-likelihood function is used to represent the similarity between the integrated descriptors x and y, and the similarity evaluation of images A and B is obtained by normalizing them using formula (10):

[0103]

[0104] max=-log P(x0,y0) (9)

[0105]

[0106] The similarity between the integrated descriptors x and y can be obtained from formula (7). In formula (8), x0 and y0 are the integrated descriptors obtained after matching the image blocks of the pure white image and the pure black image, respectively. The calculated max is the theoretical upper limit of formula (7). Formula (9) normalizes the similarity value obtained by formula (7) to the range of [0,1]. The higher the value, the more similar the target image B is to the reference image A.

[0107] like Figure 7 and Figure 8 As shown, S3, a dual-reference deformation field evaluation model based on unsupervised contrastive learning is used to perform motion analysis on adjacent images in the scene segment to obtain motion analysis results.

[0108] Specifically, the dual-reference deformation field evaluation model based on unsupervised contrastive learning includes:

[0109] A feature extraction module configured to extract texture features of any two adjacent image blocks, a first texture feature and a second texture feature;

[0110] A feature fusion module configured to fuse the first texture feature and the second texture feature to obtain a fused feature;

[0111] A spatial transformation network module is configured to deform the floating image toward the reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks are reference images and floating images to each other;

[0112] An optimization module is configured to optimize parameters of a dual-reference deformation field evaluation model for unsupervised contrastive learning through dual-reference constraints and a loss function.

[0113] This application adopts a dual-reference deformation evaluation method to make the two images to be registered refer to each other and jointly affect the adjustment of network parameters, thus increasing the dual-reference The constraints of the deformation registration are improved, making the displacement field evaluation results more accurate and reliable.

[0114] Furthermore, the formula of the loss function is:

[0115]

[0116] Where L(R1, R2, φ) represents the loss function, R1 represents the first image block, R2 represents the second image block adjacent to the first image block, φ represents the deformation field, R represents any image in R1 or R2, and w R represents the weight of each deformation reference item, represents the similarity loss term, λ represents the regularization term coefficient, represents the regularization term of the deformation field, p represents the pixel position in the deformation field, Ω represents the domain of the deformation field, represents the local differential of the deformation field at position p, and ||·||2 represents the L2 norm.

[0117] like Fig. 9 and Fig.10 As shown, S4, the motion displacement and displacement direction between images are calculated according to the motion analysis results, the motion pattern of the small intestine is identified and classified based on the motion displacement and displacement direction, and the key frame sequence of the small intestine movement is extracted according to the motion analysis results.

[0118] In an embodiment of the present application, the motion analysis result includes a dual-channel displacement of the image block and an HSV color image after motion vector transformation, and step S4 specifically includes:

[0119] Extract dual-channel displacement information and calculate average displacement and direction statistics;

[0120] Use deep convolutional neural networks to extract deep features, analyze the movement characteristics of the small intestine, and classify the movement patterns into contraction, relaxation, and others;

[0121] Filter the output motion pattern sequence and draw the contraction and relaxation curve;

[0122] When the average pixel displacement is large and the displacement direction is multi-directionally distributed, and / or the cumulative displacement in the same scene segment exceeds a preset threshold, the corresponding image is extracted as a key frame.

[0123] It can be understood that the HSV color image after the motion vector transformation is a three-channel image, that is, the displacement vector is mapped into an HSV image, which is conducive to the classification of the movement pattern of the small intestine.

[0124] In a preferred embodiment of the present application, an improved VGG16 network is used to extract deep features.

[0125] It is understandable that other deep convolutional neural networks may also be used for depth extraction, and this application does not impose any specific limitations on this.

[0126] Furthermore, by calculating the average pixel displacement, the displacement angle value of each pixel is calculated and the displacement direction is counted, and the calculation formula is:

[0127]

[0128] Where d represents the average pixel displacement, N represents the total number of pixels involved in the calculation, Δx represents the horizontal displacement of the pixel, and Δy represents the movement distance of the pixel in the x-axis direction;

[0129] like Figure 6As shown, in one embodiment of the present application, one contraction and one relaxation are recorded as one cycle, and the movement state of the small intestine is analyzed by calculating the small intestine contraction frequency, wherein the calculation formula of the small intestine contraction frequency is:

[0130]

[0131] Among them, average-number represents the frequency of small intestine contraction.

[0132] In one embodiment of the present application, the improved VGG16 network includes six block structures, each of which is composed of a convolution layer and a pooling layer, and the number of channels in each block structure is the same. Multiple small convolution kernels are stacked to replace large convolution kernels to obtain the same receptive field, while reducing the number of network parameters and deepening the network depth.

[0133] like Fig.11 As shown in FIG. 1 , the six block structures include the first block structure, the second block structure, the third block structure, the fourth block structure, the fifth block structure and the sixth block structure, wherein the first block structure includes two 240×240×64 convolutional layers + activation functions (Convolution+ReLU), the second block structure includes a 120×120×128 maximum pooling unit and two 120×120×128 convolutional layers + activation functions (Convolution+ReLU), the third block structure includes a 60×60×256 maximum pooling layer and three 60×60×256 convolutional layers. The fourth block structure includes a 30×30×512 maximum pooling layer and three 303×0×512 convolutional layers + activation function (Convolution+ReLU), the fifth block structure includes a 15×15×512 maximum pooling layer, a 7×7×512 maximum pooling layer and three 15×15×512 convolutional layers + activation function (Convolution+ReLU), and the sixth block structure includes a 1×1×4096 fully connected layer + activation function (Full Connected+ReLU), a 1×1×4096 batch normalization (BatchNorm) layer, a 1×1×2048 fully connected layer + activation function (Full Connected+ReLU) and a 1×1×2048 batch normalization (BatchNorm) layer, a 1×1×3 volume fully connected layer + activation function (Full Connected+ReLU) and a 1×1×3 batch normalization (BatchNorm) layer.

[0134] Improve the fully connected layer of the original VGG16 network and transform it into a classifier structure to automatically identify the motion mode type corresponding to the input optical flow field image;

[0135] The improved VGG16 network can extract and classify features of the input displacement field image, thereby realizing automatic recognition of the small intestine movement pattern.

[0136] This application achieves a non-invasive assessment of the small intestine's motility system by performing scene segmentation and displacement field analysis on capsule endoscopy image sequences. It can accurately identify the movement pattern of the small intestine and extract key frame sequences, providing an objective and reliable basis for clinical diagnosis.

[0137] The present invention provides a device for analyzing a power system based on an endoscopic image sequence, using the method described above, comprising:

[0138] A first image segmentation module is configured to collect an image sequence in the small intestine through a wireless capsule endoscope and divide the image sequence into a plurality of image blocks at equal intervals;

[0139] A second image segmentation module is configured to use a SiameseCF network to extract and match features of each image block, calculate the similarity between adjacent image blocks, and segment the image sequence according to a similarity threshold to obtain multiple scene segments;

[0140] A motion analysis module, which is configured to use a dual-reference deformation field evaluation model based on unsupervised contrastive learning to perform motion analysis on adjacent images in a scene segment to obtain a motion analysis result;

[0141] The dynamic evaluation module is configured to calculate the motion displacement and displacement direction between images according to the motion analysis results, identify and classify the motion pattern of the small intestine based on the motion displacement and displacement direction, and extract the key frame sequence of the small intestine movement according to the motion analysis results.

[0142] The present invention provides an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus, wherein the processor, memory and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method described above.

[0143] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method described above.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for analyzing the small intestinal dynamic system based on endoscopic image sequences, characterized in that: The following steps are involved: S1, collecting an image sequence of the small intestine through a wireless capsule endoscope, and dividing the images in the image sequence into a plurality of image blocks at equal intervals; S2, use SiameseCF network to extract and match features of each image block, calculate the similarity between adjacent image blocks, segment the image sequence according to the similarity threshold, and obtain multiple scene segments; S3, using a dual-reference deformation field evaluation model based on unsupervised contrastive learning to perform motion analysis on adjacent images within the scene segment to obtain motion analysis results; S4. Calculate the motion displacement and displacement direction between images according to the motion analysis results, identify and classify the motion pattern of the small intestine based on the motion displacement and displacement direction, and extract the key frame sequence of the small intestine movement according to the motion analysis results.

2. A method for analyzing the small intestinal dynamics system based on endoscopic image sequences as claimed in claim 1, characterized in that: The dual-reference deformation field evaluation model based on unsupervised contrastive learning includes: A feature extraction module configured to extract texture features of any two adjacent image blocks, a first texture feature and a second texture feature; A feature fusion module configured to fuse the first texture feature and the second texture feature to obtain a fused feature; A spatial transformation network module is configured to deform the floating image toward the reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks are reference images and floating images to each other; An optimization module is configured to optimize parameters of a dual-reference deformation field evaluation model for unsupervised contrastive learning through dual-reference constraints and a loss function.

3. A method for analyzing the small intestinal dynamic system based on endoscopic image sequences as claimed in claim 2, characterized in that: The formula of the loss function is: Where L(R1, R2, φ) represents the loss function, R1 represents the first image block, R2 represents the second image block adjacent to the first image block, φ represents the deformation field, R represents any image in R1 or R2, and w R represents the weight of each deformation reference item, represents the similarity loss term, λ represents the regularization term coefficient, represents the regularization term of the deformation field, p represents the pixel position in the deformation field, Ω represents the domain of the deformation field, represents the local differential of the deformation field at position p, and ||·||2 represents the L2 norm.

4. The method for analyzing the small intestinal dynamic system based on endoscopic image sequence according to claim 1, characterized in that: The motion analysis result includes the dual-channel displacement of the image block and the HSV color image after the motion vector transformation. Step S4 specifically includes: Extract dual-channel displacement information and calculate average displacement and direction statistics; Use deep convolutional neural networks to extract deep features, analyze the movement characteristics of the small intestine, and classify the movement patterns into contraction, relaxation, and others; Filter the output motion pattern sequence and draw the contraction and relaxation curve; When the average pixel displacement is large and the displacement direction is multi-directionally distributed, and / or the cumulative displacement in the same scene segment exceeds a preset threshold, the corresponding image is extracted as a key frame.

5. The method for analyzing the small intestinal dynamic system based on endoscopic image sequence according to claim 4, characterized in that: The completion of one contraction and one relaxation is recorded as one cycle, and the movement state of the small intestine is analyzed by calculating the small intestine contraction frequency, where the calculation formula of the small intestine contraction frequency is: Among them, average-number represents the frequency of small intestine contraction.

6. The method for analyzing the small intestinal dynamic system based on endoscopic image sequences according to claim 1, characterized in that: Step S2 specifically includes: S21, taking two adjacent similar images in the image sequence as training sample pairs, constructing a sample set to train the SiameseCF network model; S22, taking the first image in the image sequence as a reference image and performing a mirroring operation, reading in the next image as a target image and performing a mirroring operation; S23, inputting the image blocks of the reference image and the image blocks of the target image as the template image and the search image into the SiameseCF network respectively, and performing matching and feature extraction between the image blocks; S24, calculating the similarity between the template image and the search image by using the Gaussian formula, and when the similarity is less than the similarity threshold, taking the target image as the beginning of a new scene segment and setting it as a new reference image; when the similarity is greater than or equal to the similarity threshold, saving the target image in the current scene segment; S25. Repeat steps S21-S24 until the entire image sequence is processed to obtain multiple scene segments.

7. A method for analyzing the small intestinal dynamics system based on endoscopic image sequences as claimed in claim 6, characterized in that: The similarity is evaluated using an integrated descriptor, specifically including: Taking the integrated descriptor of the reference image as the benchmark, after matching the image blocks through the SiameseCF network, the target image obtains the local texture description image block with the highest similarity to the reference image, as well as the location information of the image block.

8. A device for analyzing the small intestinal motility system based on endoscopic image sequences, characterized in that: The method according to any one of claims 1 to 7 comprises: A feature extraction module configured to extract texture features of any two adjacent image blocks, a first texture feature and a second texture feature; A feature fusion module configured to fuse the first texture feature and the second texture feature to obtain a fused feature; A spatial transformation network module is configured to deform the floating image toward the reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks are reference images and floating images to each other; An optimization module is configured to optimize parameters of a dual-reference deformation field evaluation model for unsupervised contrastive learning through dual-reference constraints and a loss function.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus, wherein: The processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement the method according to any one of claims 1 to 7.

Images