A method and apparatus for analyzing the small intestinal motility system based on endoscopic image sequences.

Abstract

This invention proposes a method and apparatus for analyzing the small intestinal motility system based on endoscopic image sequences, belonging to the field of computer analysis of medical images. The method includes: acquiring image sequences of the small intestine using a wireless capsule endoscope; dividing the image sequences into multiple equally spaced image blocks; using a SiameseCF network to extract and match features from each image block, calculating the similarity between adjacent image blocks, and segmenting the image blocks to obtain multiple scene segments; employing a dual-reference deformation field evaluation model based on unsupervised contrastive learning to perform motion analysis on adjacent images within each scene segment, obtaining motion analysis results; calculating the motion displacement and direction between images based on the motion analysis results, identifying and classifying the small intestinal motility patterns, and extracting keyframe sequences of small intestinal motility based on the motion analysis results. This application achieves non-invasive assessment of the small intestinal motility system, accurately identifying small intestinal motility patterns and extracting keyframe sequences.

Description

Technical Field

[0001] This invention relates to the field of computer analysis technology for medical images, and in particular to a method and apparatus for analyzing the small intestinal motility system based on endoscopic image sequences. Background Technology

[0002] Wireless Capsule Endoscopy (WCE) is a novel non-invasive gastrointestinal examination tool, also known simply as capsule endoscopy. Clinically, after the patient swallows the capsule endoscope, it moves forward under its own weight and the peristalsis of the gastrointestinal tract. The capsule endoscope imaging system automatically captures images of the digestive tract at a rate of two frames per second, and transmits the captured image data wirelessly to an external receiving terminal. During an average 7-8 hour gastrointestinal examination, tens of thousands of color images of the digestive tract can be acquired.

[0003] Normal peristalsis in the small intestine is fundamental for the effective absorption of nutrients through full contact with the intestinal wall. It is also the direct driving force for the movement of digestive contents. Motility disorders can lead to malabsorption of nutrients, causing nutritional deficiencies and other health problems. Simultaneously, small intestinal peristalsis helps mix digestive juices and chyme, promoting the action of digestive enzymes. Abnormal peristalsis may result in inadequate mixing of chyme and digestive juices, affecting digestion. Insufficient small intestinal motility can also lead to bacterial overgrowth syndrome, causing adverse symptoms such as bloating and diarrhea. Assessment of the small intestinal motility system is a key method for understanding and diagnosing small intestinal motility dysfunction. Small intestinal motility has a significant impact on digestive system function, and assessment plays a crucial role in the diagnosis and management of digestive system dysfunction.

[0004] In existing technologies, small intestinal manometry is the most widely accepted and reliable method for assessing small intestinal motility. However, manometry is an invasive measurement method that determines small intestinal motility by measuring pressure at certain points within the intestine. Although existing technologies can acquire a large number of digestive tract images using WCE (whole intestine coagulation) technology, there is a lack of effective methods for extracting and analyzing motility information from WCE image sequences.

[0005] Therefore, finding a method that can non-invasively assess small intestinal motility while improving the efficiency and accuracy of small intestinal motility assessment is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention proposes a method and apparatus for analyzing the small intestinal motility system based on endoscopic image sequence analysis, which can perform non-invasive assessment of the small intestinal motility system, accurately identify the motility pattern of the small intestine and extract key frame sequences.

[0007] The technical solution of this invention is implemented as follows:

[0008] In a first aspect, the present invention provides a method for analyzing the small intestinal motility system based on endoscopic image sequences, comprising the following steps:

[0009] S1. Acquire an image sequence of the small intestine using a wireless capsule endoscope, and divide the images in the image sequence into multiple image blocks at equal intervals;

[0010] S2. Use the SiameseCF network to extract and match features for each image block, calculate the similarity between adjacent image blocks, and segment the image sequence according to the similarity threshold to obtain multiple scene segments;

[0011] S3. A dual-reference deformation field evaluation model based on unsupervised contrastive learning is used to perform motion analysis on adjacent images within a scene segment to obtain motion analysis results.

[0012] S4. Calculate the motion displacement and direction between images based on the motion analysis results, identify and classify the movement pattern of the small intestine based on the motion displacement and direction, and extract the key frame sequence of small intestine movement based on the motion analysis results.

[0013] Based on the above technical solutions, preferably, the dual-reference deformation field evaluation model based on unsupervised contrastive learning includes:

[0014] The feature extraction module is configured to extract texture features, first texture features and second texture features, from any two adjacent image patches.

[0015] The feature fusion module is configured to fuse the first texture feature and the second texture feature to obtain the fused feature.

[0016] The spatial transformation network module is configured to deform a floating image into a reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks serve as the reference image and the floating image to each other.

[0017] The optimization module is configured to optimize the parameters of the unsupervised contrastive learning dual-reference deformation field evaluation model through dual-reference constraints and a loss function.

[0018] Based on the above technical solutions, preferably, the formula for the loss function is:

[0019]

[0020] Where L(R1,R2,φ) represents the loss function, R1 represents the first image patch, R2 represents the second image patch adjacent to the first image patch, φ represents the deformation field, R represents any image in R1 or R2, and w R This represents the weight of each deformation reference term. Let λ represent the similarity loss term, and let λ represent the regularization coefficient. Let represent the regularization term of the deformation field, p represent the pixel position in the deformation field, and Ω represent the neighborhood of the deformation field. Let ||·||2 represent the local differential of the deformation field at position p, and let ||·||2 represent the L2 norm.

[0021] Based on the above technical solutions, preferably, the motion analysis results include the dual-channel displacement of the image block and the HSV color image after displacement vector transformation, and step S4 specifically includes:

[0022] Extract dual-channel displacement information and calculate average displacement and direction statistics;

[0023] Deep convolutional neural networks were used to extract deep features to analyze the motility characteristics of the small intestine, and the motility patterns were divided into contraction, relaxation and others.

[0024] The output motion pattern sequence is filtered, and the contraction and relaxation curves are plotted.

[0025] When the average pixel displacement is large and the displacement direction is multi-directionally distributed, and / or the cumulative displacement within the same scene segment exceeds a preset threshold, the corresponding image will be extracted as a keyframe.

[0026] Based on the above technical solutions, preferably, one cycle is defined as one complete contraction and one complete relaxation. The motility of the small intestine is analyzed by calculating the small intestinal contraction frequency. The formula for calculating the small intestinal contraction frequency is as follows:

[0027]

[0028] Here, average-number represents the frequency of small intestinal contractions.

[0029] Furthermore, step S2 specifically includes:

[0030] S21. Use two adjacent similar images in the image sequence as training sample pairs to construct a sample set to train the SiameseCF network model;

[0031] S22. Use the first image in the image sequence as the reference image and perform a mirror operation. Then read in the next image as the target image and perform a mirror operation.

[0032] S23. Input the image patches of the reference image and the image patches of the target image as template images and search images into the SiameseCF network to perform image patch matching and feature extraction.

[0033] S24. Calculate the similarity between the template image and the search image using the Gaussian formula. When the similarity is less than the similarity threshold, set the target image as the beginning of a new scene segment and set it as a new reference image. When the similarity is greater than or equal to the similarity threshold, save the target image to the current scene segment.

[0034] S25. Repeat steps S21-S24 until the entire image sequence is processed, resulting in multiple scene segments.

[0035] Based on the above technical solutions, preferably, the similarity is evaluated using an integrated descriptor, specifically including:

[0036] Based on the integrated descriptor of the reference image, after matching image patches through the SiameseCF network, the target image obtains the local texture description image patch with the highest similarity to the reference image, as well as the position information of the image patch.

[0037] Secondly, the present invention provides a device for analyzing the small intestinal motility system based on endoscopic image sequence analysis, employing the method described above, including:

[0038] The feature extraction module is configured to extract texture features, first texture features and second texture features, from any two adjacent image patches.

[0039] The feature fusion module is configured to fuse the first texture feature and the second texture feature to obtain the fused feature.

[0040] The spatial transformation network module is configured to deform a floating image into a reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks serve as the reference image and the floating image to each other.

[0041] The optimization module is configured to optimize the parameters of the unsupervised contrastive learning dual-reference deformation field evaluation model through dual-reference constraints and a loss function.

[0042] Thirdly, the present invention provides an electronic device, characterized in that it comprises: at least one processor, at least one memory, a communication interface, and a bus, wherein,

[0043] The processor, memory, and communication interface communicate with each other through the bus;

[0044] The memory stores program instructions that can be executed by the processor, which calls the program instructions to implement the method described above.

[0045] Fourthly, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions that cause the computer to perform the method described above.

[0046] The method of the present invention for analyzing the small intestinal motility system based on endoscopic image sequences has the following advantages over the prior art:

[0047] (1) By performing scene segmentation and displacement field analysis on capsule endoscopy image sequences, a non-invasive assessment of the small intestinal motility system was achieved. The movement patterns of the small intestine can be accurately identified and key frame sequences can be extracted, providing an objective and reliable basis for clinical diagnosis.

[0048] (2) By adopting the dual-reference deformation evaluation method, the two images to be registered are used as references to each other and jointly affect the adjustment of network parameters. This increases the constraints of dual references, improves the accuracy of deformation registration, and makes the displacement field evaluation results more accurate and reliable.

[0049] (3) By using an integrated descriptor to evaluate the similarity between two capsule endoscopy images, the images are divided into 36 image blocks and combined with position information, which effectively expresses the texture information of different regions in the capsule endoscopy images, improves the accuracy of scene segmentation, and provides a reliable foundation for subsequent motion analysis. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention;

[0052] Figure 2 This is a flowchart of the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention;

[0053] Figure 3 This is a flowchart of scene segmentation based on SiameseCF network capsule endoscopy image sequence similarity for the method of analyzing the small intestinal motility system based on endoscopic image sequence according to the present invention.

[0054] Figure 4 This is a schematic diagram of the image patch distribution of images with different similarities to the method of analyzing the small intestinal motility system based on endoscopic image sequence according to the present invention; where (a) and (b) are scenes with similarity between the target capsule endoscopy image and the reference image, and (c) and (d) are capsule endoscopy images with low similarity to the reference image;

[0055] Figure 5This is a diagram of the SiameseCF network structure of the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention.

[0056] Figure 6 This is a ConvNet structure diagram of the SiameseCF network used in the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention.

[0057] Figure 7 This is a schematic diagram of the unsupervised dual-reference deformation field assessment of the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention.

[0058] Figure 8 This is a flowchart of the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention, which is based on unsupervised dual-reference deformation field assessment.

[0059] Figure 9 This is a schematic diagram of the small intestinal motility analysis network in the method of analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention;

[0060] Figure 10 This is a flowchart of the motion analysis and keyframe extraction process of the method for analyzing the small intestinal dynamic system based on endoscopic image sequences according to the present invention.

[0061] Figure 11 This is a schematic diagram of the modified VGG network structure of the method for analyzing the small intestinal motility system based on endoscopic image sequences according to the present invention. Detailed Implementation

[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0063] like Figure 1 and Figure 2 As shown, this invention provides a method for analyzing the small intestinal motility system based on endoscopic image sequences, comprising the following steps:

[0064] S1. Acquire image sequences of the small intestine using a wireless capsule endoscope, and divide the images in the image sequence into multiple image blocks at equal intervals.

[0065] In one embodiment of this application, a capsule endoscopy image is divided into 36 equally spaced 40×40 pixel image blocks. Each image block describes local texture information and records the position information of the image block in the image. The local texture and position information of the image blocks are combined to comprehensively express the features of a capsule endoscopy image.

[0066] S2. Use the SiameseCF network to extract and match features for each image block, calculate the similarity between adjacent image blocks, and segment the image sequence according to the similarity threshold to obtain multiple scene segments.

[0067] like Figure 3 and Figure 4 As shown, specifically, step S2 includes:

[0068] S21. Use two adjacent similar images in the image sequence as training sample pairs to construct a sample set to train the SiameseCF network model;

[0069] S22. Use the first image in the image sequence as the reference image and perform a mirror operation. Then read in the next image as the target image and perform a mirror operation.

[0070] S23. Input the image patches of the reference image and the image patches of the target image as template images and search images into the SiameseCF network to perform image patch matching and feature extraction.

[0071] S24. Calculate the similarity between the template image and the search image using the Gaussian formula. When the similarity is less than the similarity threshold, set the target image as the beginning of a new scene segment and set it as a new reference image. When the similarity is greater than or equal to the similarity threshold, save the target image to the current scene segment.

[0072] S25. Repeat steps S21-S24 until the entire image sequence is processed, resulting in multiple scene segments.

[0073] In a preferred embodiment of this application, the similarity threshold is set to 0.65.

[0074] like Figure 3 As shown, it is understandable that the SiameseCF network treats target tracking as a similarity learning process. It takes a template image x and a search image y as input, determines the position of the template image in the search image by judging the similarity between x and y, treats the feature map obtained by ConvNet from the template image as a convolution kernel, performs relevant operations on the feature map obtained by ConvNet from the search image, and outputs a score map of the best matching position.

[0075] In the SiameseCF network, the ConvNet is a fully convolutional network layer whose function is to extract features from the input image. The network structure is as follows: Figure 5 and Figure 6As shown, except for conv5, each convolutional layer is followed by a ReLU non-linear activation layer, and the pooling layer uses max-pooling. The network architecture is shown in Table 1, which gives the specific parameters of the convolutional and pooling layers, as well as the size of the input template image x and the search image y after passing through each network layer (this is just an example; the input image size can be adjusted according to the sample set image size).

[0076] Table 1

[0077]

[0078] Furthermore, the similarity is evaluated using an ensemble descriptor, specifically including:

[0079] Based on the integrated descriptor of the reference image, after matching image patches through the SiameseCF network, the target image obtains the local texture description image patch with the highest similarity to the reference image, as well as the position information of the image patch.

[0080] In one embodiment of this application, a capsule endoscopy image is represented by an integrated descriptor of 36 image blocks with relative position information, which can effectively express the texture information of different regions in the capsule endoscopy image.

[0081] This application employs an integrated descriptor to evaluate the similarity between two capsule endoscopy images, dividing the images into 36 image blocks and combining them with positional information. This effectively expresses the texture information of different regions in the capsule endoscopy images, improves the accuracy of scene segmentation, and provides a reliable foundation for subsequent motion analysis.

[0082] like Figure 4 As shown in one embodiment of this application, 10 image blocks are used to illustrate the similarity evaluation process of the integrated descriptor. Figure 4 If the target capsule endoscope images shown in (a) and (b) have similar scenes to the reference images, then the image patch location distribution and texture features of the matched integrated descriptor have a high similarity to the reference images. Figure 4 If the capsule endoscope images shown in (c) and (d) have low similarity to the reference images, then the image patch location distribution and texture features of the integrated descriptor obtained by the search have low similarity to the reference images.

[0083] Let reference image A and target image B be two capsule endoscopy images whose similarity is to be evaluated. Let the ensemble descriptors in images A and B be represented by x and y, respectively. The joint probability P(x,y) of the similarity between images A and B can be expressed by the formula:

[0084] P(x,y)=P(y|x)P(x) (1)

[0085] Where P(x,y) represents the joint probability distribution of reference image A and target image B, p(y|x) represents the probability of the occurrence of integrated descriptor y of target image B given the integrated descriptor x of reference image A, and P(x) represents the marginal probability distribution of integrated descriptor x of reference image A.

[0086] The value of P(x,y) is between 0 and 1. The closer the value is to 1, the more similar the integrated descriptors x and y are.

[0087] make Let represent the descriptor vector of the i-th image patch in x. c represents the descriptor vector of the i-th image patch in y. x and c y The center coordinates of the integrated descriptor representing images A and B. Let represent the coordinates of the i-th image patch in x. Let x represent the coordinates of the i-th image patch in y. Then the joint probability between x and y can be described as:

[0088]

[0089] in, This represents the descriptor vector of the first image patch in x. This represents the coordinate position of the first image patch in x. This represents the descriptor vector of the first image patch in y.

[0090] Assuming the two integrated descriptors x and y are relatively independent, then according to formula (1) and conditional independence, the joint probability P(x,y) can be decomposed as:

[0091]

[0092] Where n represents the total number of image patches, Let represent the conditional probability of the i-th image patch descriptor appearing in the target image B, given the i-th image patch descriptor in the reference image A. This represents the conditional probability of the i-th image patch location appearing in the target image B, given the location of the i-th image patch in the reference image A and the center coordinates of the two image ensemble descriptors. P(c) represents the probability of the i-th image patch in target image B occurring given the center coordinates of the integrated descriptor of target image B. y |c x ) represents the conditional probability of the center coordinates of the integrated descriptor of the target image B occurring given the center coordinates of the integrated descriptor of the reference image A.

[0093] Assuming that the ensemble descriptor vectors are independent of each other, and using a Gaussian distribution to model the similarity between matching block descriptors:

[0094]

[0095] In the formula, α1 is a constant, and S D It is a constant covariance matrix that determines the permissible deviation between descriptor vectors.

[0096] Since the positional relationships between image patches cannot be analyzed using formulas, for ease of calculation, it is assumed that the positions of each image patch in the image are relatively independent, and a Gaussian distribution is used to model the positional similarity of the matching patches:

[0097]

[0098] In the formula, α2 is a constant, and S L It is a constant covariance matrix that determines the allowable deviation between the relative positions of image patches.

[0099] The conditional probability P(c) in formula (3) y |c x ), Let x and y represent the similarity of the central image patches in set descriptors x and y, and the offset of each image patch in set descriptor y relative to the center coordinates, respectively. These can be obtained using formulas (6)-(7).

[0100]

[0101] Formula (6) indicates that if the similarity between the central image patches of the set descriptors x and y is higher than the threshold τ, the value is 1, and otherwise it is 0. Formula (7) indicates the degree of offset of each image patch in the set descriptor y relative to the center coordinates. The value is 0 if it exceeds the image range, and 1 otherwise.

[0102] Based on the above formula and the decomposition of the joint probability P(x,y) in formula (7), the log-likelihood function is used to represent the similarity between the integrated descriptors x and y in this experiment, and formula (10) is used to normalize it to obtain the similarity evaluation of images A and B:

[0103]

[0104] max=-log P(x0,y0) (9)

[0105]

[0106] The similarity between the ensemble descriptors x and y can be obtained from formula (7). In formula (8), x0 and y0 are the ensemble descriptors obtained after matching image blocks of pure white and pure black images, respectively. The calculated max is the theoretical upper limit of formula (7). Formula (9) normalizes the similarity value obtained from formula (7) to the range of [0,1]. The higher the value, the more similar the target image B is to the reference image A.

[0107] like Figure 7 and Figure 8 As shown in Figure S3, a dual-reference deformation field evaluation model based on unsupervised contrastive learning is used to perform motion analysis on adjacent images within the scene segment to obtain motion analysis results.

[0108] Specifically, the dual-reference deformation field evaluation model based on unsupervised contrastive learning includes:

[0109] The feature extraction module is configured to extract texture features, first texture features and second texture features, from any two adjacent image patches.

[0110] The feature fusion module is configured to fuse the first texture feature and the second texture feature to obtain the fused feature.

[0111] The spatial transformation network module is configured to deform a floating image into a reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks serve as the reference image and the floating image to each other.

[0112] The optimization module is configured to optimize the parameters of the unsupervised contrastive learning dual-reference deformation field evaluation model through dual-reference constraints and a loss function.

[0113] This application employs a dual-reference deformation evaluation method, whereby the two images to be registered serve as mutual references, jointly influencing the adjustment of network parameters, thus enhancing the dual-reference effect. The constraints improve the accuracy of deformation registration, making the displacement field assessment results more accurate and reliable.

[0114] Furthermore, the formula for the loss function is:

[0115]

[0116] Where L(R1,R2,φ) represents the loss function, R1 represents the first image patch, R2 represents the second image patch adjacent to the first image patch, φ represents the deformation field, R represents any image in R1 or R2, and w R This represents the weight of each deformation reference term. Let λ represent the similarity loss term, and let λ represent the regularization coefficient. Let represent the regularization term of the deformation field, p represent the pixel position in the deformation field, and Ω represent the neighborhood of the deformation field. Let ||·||2 represent the local differential of the deformation field at position p, and let ||·||2 represent the L2 norm.

[0117] like Figure 9 and Figure 10 As shown, S4 calculates the motion displacement and direction between images based on the motion analysis results, identifies and classifies the movement pattern of the small intestine based on the motion displacement and direction, and extracts the key frame sequence of small intestine movement based on the motion analysis results.

[0118] In one embodiment of this application, the motion analysis result includes the dual-channel displacement of the image patch and the HSV color image after motion vector transformation. Step S4 specifically includes:

[0119] Extract dual-channel displacement information and calculate average displacement and direction statistics;

[0120] Deep convolutional neural networks were used to extract deep features to analyze the motility characteristics of the small intestine, and the motility patterns were divided into contraction, relaxation and others.

[0121] The output motion pattern sequence is filtered, and the contraction and relaxation curves are plotted.

[0122] When the average pixel displacement is large and the displacement direction is multi-directionally distributed, and / or the cumulative displacement within the same scene segment exceeds a preset threshold, the corresponding image will be extracted as a keyframe.

[0123] Understandably, the HSV color image after motion vector transformation is a three-channel image, which maps the displacement vector to an HSV image, making it helpful for classifying the movement patterns of the small intestine.

[0124] In a preferred embodiment of this application, an improved VGG16 network is used to extract deep features.

[0125] It is understandable that other deep convolutional neural networks can also be used for depth extraction, and this application does not impose any specific restrictions on this.

[0126] Furthermore, by calculating the average pixel displacement, simultaneously calculating the displacement angle value of each pixel and statistically analyzing the displacement direction, the calculation formula is as follows:

[0127]

[0128] Where d represents the average pixel displacement, N represents the total number of pixels involved in the calculation, Δx represents the horizontal displacement of the pixel, and Δy represents the distance the pixel moves along the x-axis.

[0129] like Figure 6As shown in one embodiment of this application, one complete contraction and one relaxation is recorded as one cycle. The motility of the small intestine is analyzed by calculating the small intestinal contraction frequency. The formula for calculating the small intestinal contraction frequency is as follows:

[0130]

[0131] Here, average-number represents the frequency of small intestinal contractions.

[0132] In one embodiment of this application, the improved VGG16 network includes six block structures, each consisting of convolutional layers and pooling layers, with the same number of channels in each block structure. Multiple small convolutional kernels are stacked instead of large convolutional kernels to achieve the same receptive field, while reducing the number of network parameters and increasing network depth.

[0133] like Figure 11 As shown, the six-block structure includes a first block structure, a second block structure, a third block structure, a fourth block structure, a fifth block structure, and a sixth block structure. The first block structure includes two 240×240×64 convolutional layers with activation functions (Convolution+ReLU). The second block structure includes one 120×120×128 max-pooling unit and two 120×120×128 convolutional layers with activation functions (Convolution+ReLU). The third block structure includes one 60×60×256 max-pooling layer and three 60×60×256 convolutional layers. The first structure consists of a convolutional layer with ReLU activation. The fourth structure includes a 30×30×512 max-pooling layer and three 30×3×0×512 convolutional layers with ReLU activation. The fifth structure includes a 15×15×512 max-pooling layer, a 7×7×512 max-pooling layer, and three 15×15×512 convolutional layers with ReLU activation. The sixth structure includes a 1×1×4096 fully connected layer with ReLU activation. The system consists of a 1×1×4096 batch normalization (BatchNorm) layer, a 1×1×2048 fully connected layer with activation function (Full Connected+ReLU), a 1×1×2048 batch normalization (BatchNorm) layer, a 1×1×3 convolutional fully connected layer with activation function (Full Connected+ReLU), and a 1×1×3 batch normalization (BatchNorm) layer.

[0134] The fully connected layers of the original VGG16 network are improved and transformed into a classifier structure to automatically determine the motion mode type corresponding to the input optical flow field image.

[0135] The improved VGG16 network can extract and classify features from the input displacement field image, enabling automatic recognition of small intestinal motility patterns.

[0136] This application achieves non-invasive assessment of the small intestinal motility system by performing scene segmentation and displacement field analysis on capsule endoscopy image sequences. It can accurately identify the motility patterns of the small intestine and extract key frame sequences, providing an objective and reliable basis for clinical diagnosis.

[0137] This invention provides a device for a power system based on endoscopic image sequence analysis, employing the method described above, comprising:

[0138] The first image segmentation module is configured to acquire an image sequence inside the small intestine via a wireless capsule endoscope and divide the image sequence into multiple image blocks at equal intervals.

[0139] The second image segmentation module is configured to use the SiameseCF network to extract and match features for each image block, calculate the similarity between adjacent image blocks, and segment the image sequence according to the similarity threshold to obtain multiple scene segments;

[0140] The motion analysis module is configured to use a dual-reference deformation field evaluation model based on unsupervised contrastive learning to perform motion analysis on adjacent images within a scene segment and obtain motion analysis results.

[0141] The dynamic assessment module is configured to calculate the motion displacement and direction between images based on the motion analysis results, identify and classify the motion patterns of the small intestine based on the motion displacement and direction, and extract the key frame sequence of small intestine motion based on the motion analysis results.

[0142] The present invention provides an electronic device, comprising: at least one processor, at least one memory, a communication interface, and a bus, wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the method described above.

[0143] The present invention provides a computer-readable storage medium that stores computer instructions that cause the computer to perform the method described above.

[0144] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for analyzing the small intestinal motility system based on endoscopic image sequences, characterized in that, Includes the following steps: S1. Acquire an image sequence of the small intestine using a wireless capsule endoscope, and divide the images in the image sequence into multiple image blocks at equal intervals; S2. Use the SiameseCF network to extract and match features for each image block, calculate the similarity between adjacent image blocks, and segment the image sequence according to the similarity threshold to obtain multiple scene segments; S3. An unsupervised contrastive learning-based dual-reference deformation field evaluation model is adopted to perform motion analysis on adjacent images within a scene segment and obtain motion analysis results. The unsupervised contrastive learning-based dual-reference deformation field evaluation model includes an optimization module, which is configured to optimize the parameters of the unsupervised contrastive learning-based dual-reference deformation field evaluation model through dual-reference constraints and loss functions. S4. Calculate the motion displacement and direction between images based on the motion analysis results, identify and classify the movement pattern of the small intestine based on the motion displacement and direction, and extract the key frame sequence of small intestine movement based on the motion analysis results. The motion analysis results include the dual-channel displacement of the image patch and the HSV color image after motion vector transformation. Step S4 specifically includes: Extract dual-channel displacement information and calculate average displacement and direction statistics; Deep convolutional neural networks were used to extract deep features to analyze the motility characteristics of the small intestine, and the motility patterns were divided into contraction, relaxation and others. The output motion pattern sequence is filtered, and the contraction and relaxation curves are plotted. When the cumulative displacement within the same scene segment exceeds a preset threshold, the corresponding image is extracted as a keyframe.

2. The method for analyzing the small intestinal motility system based on endoscopic image sequences as described in claim 1, characterized in that, The dual-reference deformation field evaluation model based on unsupervised contrastive learning also includes: The feature extraction module is configured to extract texture features, first texture features and second texture features, from any two adjacent image patches. The feature fusion module is configured to fuse the first texture feature and the second texture feature to obtain the fused feature. The spatial transformation network module is configured to deform a floating image into a reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks serve as the reference image and the floating image to each other.

3. The method for analyzing the small intestinal motility system based on endoscopic image sequences as described in claim 2, characterized in that, The formula for the loss function is: ； ； in, Represents the loss function. Indicates the first image block. This represents the second image block adjacent to the first image block. Represents the deformation field. Represents either image in R1 or R2. This represents the weight of each deformation reference term. Represents the similarity loss term. Represents the regularization coefficient. Represents the regularization term for the deformation field. This indicates the position of a pixel in the deformation field. The domain representing the deformation field. This represents the local differential of the deformation field at position p. This represents the L2 norm.

4. The method for analyzing the small intestinal motility system based on endoscopic image sequences as described in claim 1, characterized in that, One complete contraction and one relaxation is defined as one cycle. The motility of the small intestine is analyzed by calculating the contraction frequency. The formula for calculating the contraction frequency of the small intestine is as follows: ； in, This indicates the frequency of small intestinal contractions.

5. The method for analyzing the small intestinal motility system based on endoscopic image sequences as described in claim 1, characterized in that, Step S2 specifically includes: S21. Use two adjacent similar images in the image sequence as training sample pairs to construct a sample set to train the SiameseCF network model; S22. Use the first image in the image sequence as the reference image and perform a mirror operation. Then read in the next image as the target image and perform a mirror operation. S23. Input the image patches of the reference image and the image patches of the target image as template images and search images into the SiameseCF network to perform image patch matching and feature extraction. S24. Calculate the similarity between the template image and the search image using the Gaussian formula. When the similarity is less than the similarity threshold, set the target image as the beginning of a new scene segment and set it as a new reference image. When the similarity is greater than or equal to the similarity threshold, save the target image to the current scene segment. S25. Repeat steps S21-S24 until the entire image sequence is processed, resulting in multiple scene segments.

6. The method for analyzing the small intestinal motility system based on endoscopic image sequences as described in claim 5, characterized in that, The similarity is evaluated using an integrated descriptor, specifically including: Based on the integrated descriptor of the reference image, after matching image patches through the SiameseCF network, the target image obtains the local texture description image patch with the highest similarity to the reference image, as well as the position information of the image patch.

7. A device for analyzing the small intestinal motility system based on endoscopic image sequence analysis, characterized in that, The method described in any one of claims 1-6 includes: The feature extraction module is configured to extract texture features, first texture features and second texture features, from any two adjacent image patches. The feature fusion module is configured to fuse the first texture feature and the second texture feature to obtain the fused feature. The spatial transformation network module is configured to deform a floating image into a reference image through the spatial transformation network to obtain a deformation field; wherein any two adjacent image blocks serve as the reference image and the floating image to each other. The optimization module is configured to optimize the parameters of the unsupervised contrastive learning dual-reference deformation field evaluation model through dual-reference constraints and a loss function.

8. An electronic device, characterized in that, include: At least one processor, at least one memory, a communication interface, and a bus, wherein, The processor, memory, and communication interface communicate with each other through the bus; The memory stores program instructions that can be executed by the processor, which invokes the program instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1-6.

Images