An enteroscopy capsule video ulcer fragment automatic positioning method, computer device and system

Abstract

This invention relates to an automatic localization method, computer device, and system for ulcer fragments in intestinal capsule endoscopy videos. It establishes an automatic ulcer fragment localization network based on initial temporal boundary specifications, combined with policy gradients and evaluation. The network is trained using query image-reference video pairs. The network is input into the trained network and outputs ulcer fragments in the reference video associated with the query image. A computer device is implemented based on this method. The system uses an input unit to input query image-reference video pairs, a localization unit to execute the method and locate the ulcer fragments, and an output unit to output the results. This invention uses an image of an ulcer lesion of interest from an intestinal capsule endoscopy as the query image and a long intestinal capsule endoscopy video as the reference video. It analyzes the video using reinforcement learning, refines the temporal boundary through sequential decisions using start and end markers, and accurately locates fragments related to the query image, thus completing the video relocalization task.

Description

Technical Field

[0001] This invention relates to the technical field of image or video recognition or understanding, and in particular to an automatic localization method, computer equipment and system for ulcer fragments in intestinal capsule endoscopy video. Background Technology

[0002] The small intestine is the main site of digestion and absorption, averaging 5-7 meters in length. It consists of the duodenum, jejunum, and ileum. Due to its unique structure, traditional handheld gastroscopes and colonoscopes cannot perform a complete examination of it. Therefore, capsule endoscopy, as a non-invasive technique, has become an effective method for diagnosing digestive tract diseases, especially small intestinal diseases.

[0003] Capsule endoscopy works by having the patient swallow a smart capsule with a built-in camera, allowing for real-time imaging of areas inaccessible to traditional endoscopes. However, capsule endoscopy generates 5-8 hours of continuous video, corresponding to 50,000-60,000 images. Of these, lesion images typically account for less than 1%, and the images often contain a large amount of irrelevant interference, such as air bubbles, food debris, and bile. This results in low efficiency and a heavy workload for doctors reviewing the images, often requiring repeated examinations to avoid missed or misdiagnosed cases. Traditional frame-by-frame ulcer lesion identification methods are inefficient. However, considering that ulcer lesions usually appear in consecutive video frames, the contextual relationships between these frames can be used to directly locate ulcer lesion segments from long videos, improving computational efficiency. Summary of the Invention

[0004] This invention solves the problems existing in the prior art and provides an automatic localization method, computer equipment and system for intestinal capsule endoscopy video ulcer fragments.

[0005] The technical solution adopted in this invention is an automatic localization method for ulcer fragments in intestinal capsule endoscopy video. The method establishes an automatic localization network for ulcer fragments based on initial time boundary specifications, combined with policy gradients and evaluation.

[0006] The network for automatically locating ulcer fragments was trained using query image-reference video pairs.

[0007] The network automatically locates the ulcer fragments trained on the query image-reference video pair as input, and outputs the ulcer fragments in the reference video that are associated with the query image.

[0008] Preferably, the automatic localization network for ulcer fragments includes a feature extraction module, an observation network module, and a localization module based on the A2C algorithm arranged sequentially, and a prior time constraint module is also provided between the feature extraction module and the observation network module.

[0009] Preferably, the image features and video features output by the feature extraction module are input into the prior time constraint module to obtain the frame index of the reference video with the highest feature similarity after averaging the image features. The position of this frame in the video feature sequence is used as the initial start normalization time boundary and the end normalization time boundary.

[0010] Preferably, the observation network module includes an image observation branch corresponding to the image features output by the feature extraction module, a video observation branch corresponding to the video features output by the feature extraction module, and a time observation branch corresponding to the initial normalized time boundary output by the reference video and prior time constraint module. There are associated branches between the video observation branch, the image observation branch, and the time observation branch. All branches are output after passing through a fully connected layer.

[0011] Preferably, the image observation branch includes an average pooling layer, a bidirectional gated recurrent unit, and a fully connected layer arranged sequentially;

[0012] The video observation branch includes a sequentially arranged bidirectional gated recurrent unit and an average pooling layer;

[0013] The time observation branch includes a fully connected layer that uses the reference video and the start and end normalized time boundaries as normalized time constraints to output normalized time boundary information.

[0014] The associated branch includes a feature extraction unit, an average pooling layer, and a fully connected layer arranged in sequence. The fully connected layer of the associated branch and the fully connected layer of the image observation branch are multiplied together and output. The normalized temporal boundary information and video features are input to the feature extraction unit.

[0015] The output features of the bidirectional gated loop unit of the image observation branch, the output of the associated branch, the output of the video observation branch, and the output features of the time observation branch are concatenated and then output through a fully connected layer.

[0016] Preferably, the localization module based on the A2C algorithm includes a gated loop unit and a multilayer perceptron arranged in parallel. The output of the multilayer perceptron is connected to the input of the gated loop unit. The gated loop unit is then connected to the Actor network and the Critic network through two fully connected layers to obtain the corresponding action strategy and state value.

[0017] Preferably, a time-distance regression branch is set between the multilayer perceptron and the gated loop unit to predict the distance between the location segment time position and the target time position at time step t, thereby obtaining the target position distance, and the predicted value of the target position distance by the multilayer perceptron is input into the gated loop unit.

[0018] Preferably, the ulcer fragment automatic localization network is trained with a loss function, which is associated with the loss of the localization module, the loss of the temporal distance regression branch, the loss of the temporal intersection-union regression, and the loss of the normalized temporal location regression.

[0019] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic localization method for ulcer fragments in intestinal capsule endoscopy video.

[0020] An automated localization system for intestinal capsule endoscopy video ulcer fragments includes:

[0021] One input unit is used to input a query image-reference video pair;

[0022] A positioning unit is used to execute the automatic positioning method for ulcer segments in intestinal capsule endoscopy video and to locate the ulcer segments.

[0023] An output unit is used to output the ulcer fragment in the reference video that is associated with the query image.

[0024] This invention relates to a method, computer device, and system for automatically locating ulcer fragments in intestinal capsule endoscopy videos. The method involves establishing an automatic ulcer fragment localization network based on initial time boundary specifications, combined with policy gradients and evaluation; training the network with query image-reference video pairs; inputting the query image-reference video pairs into the trained network; and outputting ulcer fragments in the reference video associated with the query image. A computer device is implemented based on this method. The system uses an input unit to input query image-reference video pairs, a localization unit to execute the method and locate the ulcer fragments, and an output unit to output the results.

[0025] The beneficial effect of this invention is that, in the localization of ulcer segments in long intestinal capsule endoscopy videos, the ulcer lesion image of interest in intestinal capsule endoscopy is used as the query image, and the long intestinal capsule endoscopy video is used as the reference video. The video is analyzed based on reinforcement learning, and the time boundary is refined through sequential decision-making of start and end markers, thereby accurately locating the segment related to the query image in the long intestinal capsule endoscopy video and completing the video re-localization (VRL) task. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention;

[0027] Figure 2 This is a schematic diagram of the automatic ulcer fragment localization network structure of the present invention;

[0028] Figure 3This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] like Figure 1 As shown, this invention relates to a method for automatically locating ulcer fragments in intestinal capsule endoscopy video, comprising the following steps:

[0031] (1) Establish an automatic ulcer fragment localization network based on initial time boundary specifications, combined with policy gradients and evaluation;

[0032] (2) The ulcer fragment autolocalization network was trained using query image-reference video pairs;

[0033] (3) Input the ulcer fragment automatic localization network after training with the query image-reference video pair, and output the ulcer fragment in the reference video associated with the query image.

[0034] The method will be explained below with reference to specific content.

[0035] (1) Establish an automatic ulcer fragment localization network based on initial time boundary specifications, combined with policy gradients and evaluation;

[0036] The technical concept of this invention is to obtain the initial time boundary by using prior time constraints based on similarity, and to train the start and end labels to learn a better strategy based on policy gradient reinforcement learning. The two anchor points (start and end labels) adaptively determine the start and end boundary positioning strategy based on state information, and gradually refine the time boundary to achieve accurate positioning of target disease segments. This achieves the purpose of temporal positioning based on query image-reference video pairs in the dataset and predicting the time boundary of segments in the reference video associated with the query image.

[0037] Based on this, such as Figure 2 As shown, the automatic localization network for ulcer fragments includes a feature extraction module, an observation network module, and a localization module based on the A2C algorithm arranged sequentially. A prior time constraint module is also provided between the feature extraction module and the observation network module.

[0038] (1-1) Feature Extraction Module

[0039] The image features and video features output by the feature extraction module are input into the prior time constraint module to obtain the frame index of the reference video with the highest feature similarity after averaging the image features. The position of this frame in the video feature sequence is used as the initial start time boundary and the end time boundary of normalization.

[0040] In this invention, the feature extraction module is a pre-trained network, such as using a ResNet34 network to extract features from the image and video in the query image-reference video pair, resulting in a query image-reference video feature pair. During the training of the feature extraction module, publicly available data can be used; this invention is based on a publicly available dataset containing 921 images, including 286 images of small intestinal ulcers, 297 images of small intestinal erosion, and 338 images of normal small intestine. In practical applications, the last fully connected layer of the ResNet34 model is removed, and features are extracted from both the query image and the reference video. The dimension of each feature is 512. The query image consists of multiple images of the same type and various morphologies of ulcers, and the reference video is a capsule endoscopy video containing ulcer lesions.

[0041] In this embodiment, the initial time boundary is determined by the prior time constraint module by searching the most similar frame, which provides prior constraints for subsequent precise positioning.

[0042] Specifically, by averaging the features of the query image, we obtain...

[0043]

[0044] in, For the extracted first Zhang queries the characteristics of images. To query the total number of images, To query the average features of an image;

[0045] Calculate the similarity between the average features of the query image and the features of each frame of the reference video.

[0046]

[0047] in, , It refers to the features of each frame of the reference video. For reference video frame rate, for The transpose of the vector S, and the index corresponding to the maximum value in the row vector S. It refers to the frame number of the video frame in the reference video that is most similar to the query image;

[0048] With index Divide by video frame rate The normalized temporal boundary of the video frame in the reference video that is most similar to the query image is obtained, denoted as . , Use the normalized time boundary of this frame. The initial normalized time boundary and the final normalized time boundary serve as the initial localization segment, i.e. ,in, The normalized time boundary for the initial location segment, This serves as the initial normalized time boundary for the initial localization segment. This is the termination normalized time boundary of the initial positioning segment.

[0049] Through the processing of the prior time constraint module, a specific frame number is directly used as the start and end boundary of the initial segment and is converted into a normalized time boundary. This initial normalized time boundary provides an effective initial search point for subsequent reinforcement learning, constraining the range of possible locations that need to be explored, which helps to improve the efficiency and accuracy of localization.

[0050] The normalized time boundary is continuously adjusted based on the initial normalized time boundary, and its essence is the prediction of the time marker. For example, if the initial normalized time boundary is [0.5, 0.5], the first decision of the anchor point is to move the initial normalized time boundary to the left by 0.02 and the final normalized time boundary to the right by 0.05. Then the normalized time boundary after the action becomes [0.48, 0.55]. When the requirements are met, the normalized time boundary is multiplied by the duration of the video (e.g., 10 minutes) to obtain the first prediction of the time marker, which is [288, 330].

[0051] (1-2) Observation Network Module

[0052] The observation network module includes an image observation branch corresponding to the image features output by the feature extraction module, a video observation branch corresponding to the video features output by the feature extraction module, and a time observation branch corresponding to the initial normalized time boundary output by the reference video and prior time constraint module. There are associated branches between the video observation branch, the image observation branch, and the time observation branch. All branches are output after passing through a fully connected layer.

[0053] In this invention, the input to the observation network module is the extracted query image-reference video pair and the obtained initial normalized time boundary. The observation network module generates observation features by fusing these information. During the process, the query image features and video features remain unchanged, while the normalized time boundary of the localized segment changes with the actions of the localization module. By continuously adjusting this boundary, the continuous image segments in the reference video that are related to the content of the query image are finally determined.

[0054] The image observation branch includes an average pooling layer, a bidirectional gated recurrent unit, and a fully connected layer arranged in sequence.

[0055] The video observation branch includes a sequentially arranged bidirectional gated recurrent unit and an average pooling layer;

[0056] The time observation branch includes a fully connected layer that uses the reference video and the start and end normalized time boundaries as normalized time constraints to output normalized time boundary information.

[0057] The associated branch includes a feature extraction unit, an average pooling layer, and a fully connected layer arranged in sequence. The fully connected layer of the associated branch and the fully connected layer of the image observation branch are multiplied together and output. The normalized temporal boundary information and video features are input to the feature extraction unit.

[0058] The output features of the bidirectional gated loop unit of the image observation branch, the output of the associated branch, the output of the video observation branch, and the output features of the time observation branch are concatenated and then output through a fully connected layer.

[0059] Specifically, in the image observation branch, the features of the query image are first processed by average pooling, and then encoded by a bidirectional gated recurrent unit (Bi-GRU) to obtain the query features. ;

[0060] In the video observation branch, the feature sequence of the reference video is encoded by a bidirectional gated recurrent unit (Bi-GRU) and then subjected to average pooling to obtain global features. ;

[0061] In the temporal observation branch, the reference video and the initial and final normalized time boundaries are used as inputs to the fully connected layer as normalized time constraints, which are then encoded as temporal boundary features. ;

[0062] In the association branch, based on the current normalized time boundary, the feature sequence of the corresponding local segment is extracted from the reference video, and local features are obtained through average pooling. Then combine it with The interactive features are obtained by multiplying the features after passing them through two independent fully connected layers. ;

[0063] Finally, the features obtained from each branch are concatenated and then fed into a fully connected layer to fuse them into the observed features. .

[0064] (1-3) Positioning module based on A2C algorithm

[0065] The localization module based on the A2C algorithm includes a gated loop unit and a multilayer perceptron arranged in parallel. The output of the multilayer perceptron is connected to the input of the gated loop unit. The gated loop unit is then connected to the Actor network and the Critic network through two fully connected layers to obtain the corresponding action strategy and state value.

[0066] A time-distance regression branch is set between the multilayer perceptron and the gated loop unit to predict the distance between the location segment's time position and the target time position at time step t, thereby obtaining the target position distance. The predicted value of the target position distance by the multilayer perceptron is then input into the gated loop unit.

[0067] In this invention, both the Actor network and the Critic network of the localization module generate observation features O from the image observation branch. t As input, a neural network is used to generate action policies and state values. The Actor executes the action, and the Critic evaluates the action value. Anchor points with the same network structure are configured at the start and end positions of the corresponding video segments. Both anchor points select a policy from a space containing seven actions: movement and no movement in the left and right directions with three different amplitudes (d1, d2, d3). The normalized time boundary of the currently located segment is then adjusted according to the policy. At each time step, the two anchor points independently update the start and end normalized time boundaries according to their respective policies to progressively refine the localization.

[0068] Specifically, use A i (i=s,e) represents an anchor pair, where A s Indicates the starting anchor point, A e Indicates the termination anchor point;

[0069] For A i (i=s,e) Design a time-distance regression branch to predict the target location distance at time step t. , that is, the distance between the location segment at time step t and the target time position, expressed as , Anchor point A i The start / end time position of the positioning segment after the action is performed at time step t, g i Indicates the start / end time position of the target segment;

[0070] Using a multilayer perceptron (MLP) to determine the target location distance The predicted value is This allows anchor points to better handle time intervals, infer appropriate movement directions and magnitudes, and formulate better strategies. For example, if... If the value is negative and the absolute value is large, then the start / end time position of the current segment is to the left of the start / end time position of the target segment and is far away. Therefore, a move strategy of moving to the right with a large magnitude is selected. When deciding which magnitude to move in which direction, it is based on the probability corresponding to each action. The action with the highest probability is selected. The action generation strategy of the Actor is actually the probability distribution of the action predicted by the network.

[0071] Define the localization segment after the action is executed at time step t. With target fragment Time intersection ratio satisfy,

[0072]

[0073] Subsequently, the observed feature O t Distance to predicted time and location After connection, the data is sent to the gate control recurrent unit (GRU), whose output is received by two different fully connected layers, which correspond to the Actor network and the Critic network, respectively. The output of the fully connected layer of the Actor network is normalized by the Softmax function and finally outputs a probability distribution covering 7 optional actions. The Critic network outputs a scalar, namely the state value function, which represents the expected cumulative reward that can be obtained by the anchor point in the current state.

[0074] t time step anchor point A i The reward obtained after performing an action is defined as follows:

[0075]

[0076] in, This indicates that at time step t, anchor point A... i The action performed, where trigger indicates a non-movement action. At time step t, anchor point A i The basic reward for performing the action, This indicates that at time step t, anchor point A... i The potential reward for performing the action, It is a reward specifically designed to trigger the action.

[0077] Define time and location conditions ,in, and These represent the start and end times of the location segment after t time steps, respectively, and the base reward. The reward is defined based on whether the time boundary after the anchor point's movement action exceeds the limit. When the time boundary after the anchor point's action meets the above conditions, a non-negative reward is given; otherwise, a penalty is imposed.

[0078]

[0079] in, Let t be the distance between the location segment's time position and the target time position at time step t. The distance between the location segment's time position after the action is executed at time step t and the target time position. This indicates the start or end time position of the segment after the action at time step t, and p indicates that the penalty for the action going out of bounds is set to a negative value;

[0080] The initial reward may be negative, therefore a potential reward is designed. A positive value encourages anchor point exploration actions, i.e. γ is the discount factor;

[0081] Rewards for not performing an action Based on time, location, and distance Is it small enough to be set, i.e. ,in, It is a threshold. A positive value is triggered when the distance between the location segment position after the action is executed at time step t and the target time position is less than a threshold. Otherwise, the reward will be negative; during the testing process of this invention, ∈[0,1], The effect is optimal when the value is 1;

[0082] Set a certain time step This requires the anchor point to reach the target position through multiple boundary movements. Therefore, it is necessary to consider not only the reward of the current action but also the reward of future actions, accumulating the reward. Defined as,

[0083]

[0084] Indicates the anchor point at the maximum time step. The state value after the action is performed. For the observed features after the action is performed at the maximum time step, anchor point The parameters of the Critic network, This is a discount factor used to control the impact of future rewards on cumulative rewards.

[0085] (2) The ulcer fragment autolocalization network was trained using query image-reference video pairs;

[0086] In this invention, a dedicated dataset for locating ulcer lesion fragments from capsule endoscopy videos of the gastrointestinal tract is constructed. Each sample in this dataset includes query images (several images of ulcer lesions), reference videos (long videos containing ulcer lesions), and time annotations (the time interval in which the ulcer lesion appears in the reference video, including start and end times). The dataset is divided into training and testing sets in a 3:1 ratio. The query images are from publicly available datasets, selecting 20 images of small intestinal ulcers of different morphologies from 10 patients. The reference videos are from capsule endoscopy videos of 12 patients from the same hospital, from which 243 video fragments containing ulcers are manually cropped, and each fragment is labeled with the start and end times of the ulcer lesion. In the task of locating gastrointestinal ulcer lesion fragments, each set of data consists of a query image and a reference video pair. The query images are fixed as the aforementioned 20 ulcer images, which, after being paired with different reference videos, are divided into training and testing sets in a 3:1 ratio: the training set contains 183 query image-reference video pairs, and the testing set contains 60 query image-reference video pairs.

[0087] The network for automatic localization of ulcer fragments is trained using a loss function, which is associated with the loss of the localization module, the loss of the temporal distance regression branch, the loss of the temporal intersection-union regression, and the loss of the normalized temporal location regression.

[0088] The model is trained using a Temporal Boundary Net (TB-Net), which includes an observation network module and a localization module based on the A2C algorithm. Features and initial temporal boundaries are used as training data, and the loss function L is defined as follows:

[0089]

[0090] in, , and As a weighting factor;

[0091] Let the loss function of the Actor network be...

[0092]

[0093] in, Indicates anchor point The strategy is the probability distribution of each action. Indicates the anchor point at time step t. The action performed This represents the observational characteristics at time step t. Indicates anchor point The parameters of the Actor network, regarding the anchor points Advantage function It measures the quality of performing an action in a given state relative to the average situation. Indicates cumulative rewards. Indicates the anchor point at time step t. The state value after the action is performed. The entropy of action policy is used to increase the diversity of actions. This is the weighting factor for the policy entropy;

[0094] Let be the loss function of the Critic network.

[0095]

[0096] The loss is defined for the time distance regression branch.

[0097]

[0098] in, This indicates that time distance regression is only considered when the intersection-union ratio (IoU) between the localized segment and the target segment at the previous time step is greater than or equal to 0.4. ;

[0099] Let be the loss function for time intersection-over-union (tIoU) regression.

[0100]

[0101] This represents the time intersection-union ratio between the localization segment and the target segment before the action is executed at time step t. This represents the predicted temporal intersection-union ratio (CIU) between the localization segment and the target segment before the action is executed at time step t.

[0102] The loss function for normalized time-location regression is...

[0103]

[0104] in, This means that position regression is only considered when the intersection-over-union (IoU) ratio between the localized segment and the ground truth segment at the previous time step is greater than 0.4. For the target start time position The predicted value, For the target termination time position The predicted value.

[0105] The obtained features and the initial time boundary are fused into observation features, with anchor points... The system receives observed features, performs actions, and evaluates their value. The environment then updates its state (i.e., the temporal boundary of the localization segment) and calculates the immediate reward for the action based on the reward function. This reward measures the effectiveness of the action and guides the optimization of the strategy. Based on the strategy, feedback is provided to the environment after the decision. The initial and final normalized time boundaries shift according to the action, and the system continues to iterate and participate in the calculation of the loss function. Through multiple iterations, the time boundaries are gradually adjusted and approximate the true value. Finally, the model is trained by minimizing the loss function to obtain the optimal model.

[0106] The same preprocessing was applied to the test dataset, which was then fed into the trained model to evaluate accuracy.

[0107] Treating the query image-reference video pair as a sample, if the temporal intersection ratio (tIoU) between the predicted localization segment and the target segment of each sample is not lower than the judgment threshold m, the localization is judged as correct; otherwise, the localization is judged as incorrect. The localization accuracy of each sample is... Defined as ,in, Let m represent the i-th sample and m be the decision threshold. It is the time intersection-union ratio of the i-th sample;

[0108] The evaluation metric for the matching degree between the predicted localized segment and the target segment is the mean accuracy (mAP), which represents the proportion of correctly localized samples out of all test samples.

[0109] (3) Input the ulcer fragment automatic localization network after training with the query image-reference video pair, and output the ulcer fragment in the reference video associated with the query image.

[0110] The following is a specific experimental example. The experiment used a workstation configured with an Intel(R) Xeon(R) CPU E5-2678v3 @ 2.50GHz (2 processors), 64GB of RAM, Windows 10 operating system, and four NVIDIA GeForce GTX 1080 Ti graphics cards. The CUDA version was 12.6, and the model was implemented using the PyTorch deep learning framework, version 2.4.0+cu118, and Python version 3.10. During training, the linear layer weights were initialized using a Xavier normal distribution, and the Adam algorithm was used to optimize the overall parameters. The learning rate was set to 0.001, and linear warm-up and inverse square root adjustment were used to adjust the learning rate. The model converged after 50 epochs.

[0111] Prediction was performed on the test set, with the threshold m for the time intersection-union ratio set to 0.4, 0.5, and 0.7. The default loss weights were all 1, the discount factor γ was 0.4, the maximum time step T was 10, the threshold τ was 0.05, the penalty term p was -0.8, the initial learning rate was 0.001, and the three movement amplitudes d1, d2, and d3 were 0.16, 0.05, and 0.01, respectively. The epoch was 50, and the batch size was 1. The weight parameters λ1, λ2, and λ3, the discount factor γ, and the penalty term p were controlled differently. The experimental results are shown in Tables 1 to 5.

[0112] Table 1. Test results of average accuracy mAP when the control weight parameter λ1 changes.

[0113]

[0114] Table 1 shows the test results when the weight λ1 of the Critic network loss changes. When the weight λ2 is smaller, the average accuracy decreases at each decision threshold, indicating that the accuracy of the critic's prediction of the state value is more important than other predictions.

[0115] Table 2 Test results of average accuracy mAP when the control weight parameter λ2 changes.

[0116]

[0117] Table 2 shows that the accuracy is higher when λ2 is moderate, indicating that the time intersection ratio regression helps the anchor point learn more representative state features, but appropriate weight values ​​should be selected.

[0118] Table 3 Test results of average accuracy mAP when the control weight parameter λ3 varies.

[0119]

[0120] Table 3 shows that normalized time-location regression is not very helpful for anchor point learning optimal strategies;

[0121] Table 4. Test results of average accuracy mAP when controlling for changes in discount factor γ.

[0122]

[0123] Table 4 shows that an excessively small discount factor will make the cumulative reward almost unaffected by future rewards, and the anchor point will lack long-term planning ability. An excessively large discount factor will make it more difficult to estimate the cumulative reward and the model will converge slowly. A moderate discount factor can better coordinate long-term vision and stability.

[0124] Table 5 Test results of average accuracy mAP when the control penalty term p changes.

[0125]

[0126] Table 5 shows that the larger the boundary violation penalty at the starting and ending boundaries, the more helpful it is to select the appropriate direction of movement for the anchor point.

[0127] As can be seen from Tables 1 to 5, under different parameters, the method of the present invention can achieve a high accuracy rate in locating disease segments in intestinal capsule endoscopy videos.

[0128] The present invention also relates to a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method for automatic localization of ulcer fragments in intestinal capsule endoscopy video.

[0129] like Figure 3 As shown, the present invention also relates to an automatic localization system for ulcer fragments in intestinal capsule endoscopy video, comprising:

[0130] One input unit is used to input a query image-reference video pair;

[0131] A positioning unit is used to execute the automatic positioning method for ulcer segments in intestinal capsule endoscopy video and to locate the ulcer segments.

[0132] An output unit is used to output the ulcer fragment in the reference video that is associated with the query image.

[0133] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0134] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0138] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for automatically locating ulcer fragments in intestinal capsule endoscopy video, characterized in that: The method establishes an automatic ulcer fragment localization network based on initial time boundary specifications, combined with policy gradients and evaluation. It includes a feature extraction module, an observation network module, and a localization module based on the A2C algorithm, arranged sequentially. A prior time constraint module is also provided between the feature extraction module and the observation network module. The image features and video features output by the feature extraction module are input into the prior time constraint module to obtain the frame index of the reference video with the highest feature similarity after averaging with the image features. The position of this frame in the video feature sequence is used as the initial start normalization time boundary and the end normalization time boundary. The observation network module includes an image observation branch corresponding to the image features output by the feature extraction module, a video observation branch corresponding to the video features output by the feature extraction module, and a time observation branch corresponding to the initial normalized time boundary output by the reference video and prior time constraint module. There are associated branches between the video observation branch, the image observation branch, and the time observation branch. The image observation branch includes an average pooling layer, a bidirectional gated recurrent unit, and a fully connected layer arranged in sequence. The video observation branch includes a sequentially arranged bidirectional gated recurrent unit and an average pooling layer; The time observation branch includes a fully connected layer that uses the reference video and the start and end normalized time boundaries as normalized time constraints to output normalized time boundary information. The associated branch includes a feature extraction unit, an average pooling layer, and a fully connected layer arranged in sequence. The fully connected layer of the associated branch and the fully connected layer of the image observation branch are multiplied together and output. The normalized temporal boundary information and video features are input to the feature extraction unit. The output features of the bidirectional gated loop unit of the image observation branch, the output of the correlation branch, the output of the video observation branch, and the output features of the time observation branch are concatenated and then output after passing through a fully connected layer; The network for automatically locating ulcer fragments was trained using query image-reference video pairs. The network automatically locates the ulcer fragments trained on the query image-reference video pair as input, and outputs the ulcer fragments in the reference video that are associated with the query image.

2. The method for automatic localization of ulcer fragments in intestinal capsule endoscopy video according to claim 1, characterized in that: The localization module based on the A2C algorithm includes a gated loop unit and a multilayer perceptron arranged in parallel. The output of the multilayer perceptron is connected to the input of the gated loop unit. The gated loop unit is then connected to the Actor network and the Critic network through two fully connected layers to obtain the corresponding action strategy and state value.

3. The method for automatic localization of ulcer fragments in intestinal capsule endoscopy video according to claim 2, characterized in that: A time-distance regression branch is set between the multilayer perceptron and the gated loop unit to predict the distance between the location segment's time position and the target time position at time step t, thereby obtaining the target position distance. The predicted value of the target position distance by the multilayer perceptron is then input into the gated loop unit.

4. The method for automatic localization of ulcer fragments in intestinal capsule endoscopy video according to claim 3, characterized in that: The network for automatic localization of ulcer fragments is trained using a loss function, which is associated with the loss of the localization module, the loss of the temporal distance regression branch, the loss of the temporal intersection-union regression, and the loss of the normalized temporal location regression.

5. A computer device, characterized in that: The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic localization method for intestinal capsule endoscopy video ulcer fragments according to any one of claims 1 to 4.

6. An automatic localization system for ulcer fragments in intestinal capsule endoscopy video, characterized in that: include: One input unit is used to input a query image-reference video pair; A positioning unit is used to execute the automatic positioning method for ulcer fragments in intestinal capsule endoscopy video as described in any one of claims 1 to 4 and to locate the ulcer fragments. An output unit is used to output the ulcer fragment in the reference video that is associated with the query image.

Images