WCE image scene classification method, device and equipment based on reinforcement learning

Abstract

The invention provides a WCE image scene classification method, device and equipment based on reinforcement learning, and relates to the technical field of computer analysis of medical images, and the method comprises the steps: carrying out the feature extraction of each current image frame in all current capsule endoscopy images, and obtaining a feature sequence of each current image frame, inputting all the current image frame feature sequences to a preset recurrent neural network model to obtain a first probability prediction result of each current image frame; performing image processing on the secondary classification image frame to obtain a processed image frame, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame; inputting the enhanced image frames to a preset recurrent neural network model to obtain a second probability prediction result of each enhanced image frame; and determining a target prediction result according to the first probability prediction result and the second probability prediction result. According to the invention, the reading time of the WCE can be reduced, the workload is reduced, and the classification precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer analysis of medical images, and in particular, to a method, device, and equipment for classifying WCE image scenes based on reinforcement learning. Background Art

[0002] Wireless Capsule Endoscopy (WCE) is a new type of non-invasive gastrointestinal examination tool, also known as capsule endoscopy or wireless capsule endoscope. Clinically, after a patient swallows the capsule endoscope, it moves forward under its own gravity and the peristalsis of the gastrointestinal tract, and automatically captures the entire digestive tract scene at a speed of two frames per second through the capsule endoscope imaging system. The captured image data is transmitted to an external receiving terminal in a wireless transmission manner. During an average 7-8-hour digestive tract examination, tens of thousands of color digestive tract scene images can be recorded. The human digestive system consists of a series of different organs, including the stomach, small intestine, and large intestine, and the small intestine is further divided into the duodenum, jejunum, and ileum. Therefore, each WCE image contains the organ scenes of the digestive tract.

[0003] Due to the shooting speed of the capsule endoscope and the peristalsis characteristics of the digestive tract scene, the image acquisition time for a single case is long and the number of image sequences is huge. Screening out images containing suspicious lesions from the capsule endoscope image data requires even experienced clinicians to spend 3-5 hours concentrating on accurately and error-free reading of a single case, which is a task-intensive and rather boring job. Therefore, automatically classifying the stomach, small intestine, and large intestine can reduce the reading time of WCE and relieve the work burden. However, the classification accuracy of existing classification algorithms is insufficient, resulting in classification errors, and there is no good solution strategy for the generated classification errors, leading to poor classification prediction ability for image scenes. Currently, there is no technical solution that can solve the above technical problems, and there is no method, device, and equipment for classifying WCE image scenes based on reinforcement learning. Summary of the Invention

[0004] The present invention provides a method, device, and equipment for classifying WCE image scenes based on reinforcement learning, which uses a framework of deep learning combined with hybrid reinforcement learning to efficiently and accurately classify each scene of capsule endoscope images.

[0005] In a first aspect, the present invention provides a method for classifying WCE image scenes based on reinforcement learning, including:

[0006] Feature extraction is performed on each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model to obtain a feature sequence for each current image frame. All the feature sequences of the current image frames are input into a preset recurrent neural network model to obtain a first probability prediction result for each current image frame output by the preset recurrent neural network model;

[0007] All current image frames with the first probability prediction result less than a preset threshold are determined as secondary classification image frames. For any secondary classification image frame, image processing is performed on the secondary classification image frame to obtain a processed image frame. The Q-Learning algorithm is used to select a target action, and image enhancement is performed on the processed image frame according to the target action to obtain an enhanced image frame;

[0008] The enhanced image frame is input into the preset recurrent neural network model to obtain a second probability prediction result for each enhanced image frame output by the preset recurrent neural network model;

[0009] A target prediction result for each current image frame in all current capsule endoscopy images is determined according to the first probability prediction result and the second probability prediction result;

[0010] The preset recurrent neural network model is determined by inputting a feature sequence of sample image frames into an initial recurrent neural network model to obtain a sample probability prediction result, sampling sample actions by using a polynomial distribution, continuously updating a first Q value by using the sample actions and states, obtaining a maximum expected cumulative return and a cross-entropy loss by using the sample action corresponding to the maximum first Q value and a reward value, and updating the model parameters corresponding to the initial recurrent neural network model.

[0011] According to the method for classifying WCE image scenes based on reinforcement learning provided by the present invention, before performing feature extraction on each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model, the method further includes:

[0012] Using a capsule endoscope to obtain original capsule endoscopy images;

[0013] Performing stretching and compression processing on the edges of each original capsule endoscopy image to obtain all current capsule endoscopy images.

[0014] According to the method for classifying WCE image scenes based on reinforcement learning provided by the present invention, before inputting all the feature sequences of the current image frames into a preset recurrent neural network model, the method further includes:

[0015] Performing feature extraction on all sample image frames by using a preset convolutional neural network model to obtain a feature sequence of sample image frames, and inputting the feature sequence of sample image frames into an initial recurrent neural network model to obtain a sample probability prediction result;

[0016] Sample actions are sampled using the multinomial distribution to obtain the sample results corresponding to the sample actions. The current reward value is obtained using the sample label results and the sample results corresponding to the sample actions. The current state is determined based on the current reward value and the reward value before sampling, and the first Q-value is continuously updated using the current state;

[0017] The expected cumulative return is determined using the action corresponding to the maximum first Q-value and the reward value. Gradient calculation is performed on the expected cumulative return, and the model parameters corresponding to the initial recurrent neural network model are updated using a preset formula to obtain the target model parameters. The preset recurrent neural network model is determined based on the target model parameters.

[0018] According to the WCE image scene classification method based on reinforcement learning provided by the present invention, the use of a preset convolutional neural network model to extract features from all sample image frames to obtain a sample image frame feature sequence includes:

[0019] Using a preset convolutional neural network model to extract features from all sample image frames to obtain sample extraction features;

[0020] For each sample image frame, it is stored in the HDF5 format, and the patient name, total number of frames, and sample label results corresponding to the sample extraction features of the case are recorded;

[0021] Among them, the sample image frame feature sequence is V = {X1, X2, …, X T}, where X t represents the feature vector of the t-th frame, T is the total number of frames, and V represents the case.

[0022] According to the WCE image scene classification method based on reinforcement learning provided by the present invention, the updating of the model parameters corresponding to the initial recurrent neural network model using a preset formula to obtain the target model parameters includes:

[0023]

[0024] Among them, α is the learning rate, β1 and β2 are hyperparameters for balancing weights, L CE is the cross-entropy loss function, L weight is the regularization term, J is the expected cumulative return, represents the overall derivative of the parameters θ in (-J + β1L CE + β2L weight ) to obtain the gradient.

[0025] According to the WCE image scene classification method based on reinforcement learning provided by the present invention, the first probability prediction result is the prediction probability of different scene classifications, and the scene classifications include the stomach, duodenum, jejunum, ileum, and large intestine;

[0026] Determining all current image frames with the first probability prediction result less than a preset threshold includes:

[0027] For any current image frame, when the prediction probability of each scene classification in the first probability prediction result is less than the preset threshold, determining the current image frame as a secondary classification image frame;

[0028] Traverse all current image frames until all secondary classification image frames are determined.

[0029] According to the WCE image scene classification method based on reinforcement learning provided by the present invention, performing image processing on the secondary classification image frames includes rotating the image and scaling the image;

[0030] Selecting a target action using the Q-Learning algorithm, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame, including:

[0031] For any processed image frame, using a preset convolutional neural network model to extract features of the processed image frame to obtain processed image features, and calculating the first standard deviation of the current processed image frame;

[0032] Calculating the second standard deviation of the secondary classification image frame corresponding to the current processed image frame, and determining a target reward value according to the first standard deviation and the second standard deviation;

[0033] Updating the second Q value according to the target reward value, and processing the secondary classification image frame using the action corresponding to the maximum second Q value to obtain an enhanced image frame;

[0034] Traverse all processed image frames to obtain all enhanced image frames.

[0035] According to the WCE image scene classification method based on reinforcement learning provided by the present invention, determining the target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result includes:

[0036] For any enhanced image frame, when the probability prediction result is greater than the preset threshold, determining the second probability prediction result corresponding to the enhanced image frame as the final prediction result of the enhanced image frame, otherwise, determining the first probability prediction result corresponding to the enhanced image frame as the final prediction result of the enhanced image frame;

[0037] Traverse all enhanced image frames, determine the final prediction result corresponding to each enhanced image frame, determine all current image frames whose first probability prediction result is greater than or equal to the preset threshold as unenhanced image frames, and determine the first probability prediction result corresponding to the unenhanced image frame as the final prediction result of the unenhanced image frame;

[0038] According to the final prediction result of the enhanced image frame and the final prediction result of the unenhanced image frame, determine the target prediction result of each current image frame in all current capsule endoscopy images.

[0039] In a second aspect, a WCE image scene classification device based on reinforcement learning is provided, including:

[0040] An input unit, which is used to extract features from each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model, obtain each current image frame feature sequence, and input all current image frame feature sequences into a preset recurrent neural network model to obtain the first probability prediction result of each current image frame output by the preset recurrent neural network model;

[0041] An image processing unit, which is used to determine all current image frames with a first probability prediction result less than the preset threshold as secondary classification image frames. For any secondary classification image frame, perform image processing on the secondary classification image frame to obtain a processed image frame, select a target action by using the Q-Learning algorithm, and perform image enhancement on the processed image frame according to the target action to obtain an enhanced image frame;

[0042] An output unit, which is used to input the enhanced image frame into the preset recurrent neural network model to obtain the second probability prediction result of each enhanced image frame output by the preset recurrent neural network model;

[0043] A determination unit, which is used to determine the target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result;

[0044] The preset recurrent neural network model is determined by inputting a sample image frame feature sequence into an initial recurrent neural network model to obtain a sample probability prediction result, sampling sample actions by using a polynomial distribution, continuously updating the first Q value by using the sample actions and states, obtaining the maximum expected cumulative return and cross-entropy loss by using the sample action corresponding to the maximum first Q value and the reward value, and updating the model parameters corresponding to the initial recurrent neural network model.

[0045] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for classifying WCE image scenes based on reinforcement learning is implemented.

[0046] The present invention can reduce the reading time of WCE and lighten the workload by automatically classifying the stomach, small intestine, and large intestine. At the same time, it accurately locates the target scene, which is of great significance to the automatic detection algorithm of images based on deep learning and helps to improve the detection accuracy and efficiency. The present invention focuses on the classification of scenes such as the stomach, duodenum, jejunum, ileum, and large intestine in the capsule endoscopy image sequence, and proposes a framework combining deep learning and hybrid reinforcement learning for efficiently and accurately classifying each scene of the capsule endoscopy image. A hybrid reinforcement learning is used, and two reinforcement learning methods, Monte Carlo policy gradient (REINFORCE) and action-value function (Q-Learning), are combined to classify the WCE image scenes, rather than just using a single reinforcement learning method. This not only makes up for the deficiencies of a single reinforcement learning method and improves the classification accuracy, but also evaluates the credibility of the classification results through a probability threshold method in the stage of automatically adjusting the image enhancement to correct the test results, and automatically distinguishes the images that have been correctly classified from the images with uncertain classification results. For the latter, image enhancement operations will be performed to eliminate potential classification errors. The image enhancement adjusts the enhancement strategy by estimating the action value, thereby ensuring that the quality of the image is optimized. The present invention not only optimizes the performance of the classification model using hybrid reinforcement learning, but also uses secondary prediction to improve the processing ability of uncertain classification results, and has strong practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 is one of the flow schematic diagrams of the method for classifying WCE image scenes based on reinforcement learning provided by the present invention;

[0049] Figure 2 is the second flow schematic diagram of the method for classifying WCE image scenes based on reinforcement learning provided by the present invention;

[0050] Figure 3 is the training network structure diagram provided by the present invention;

[0051] Figure 4 is the test network structure diagram provided by the present invention;

[0052] Figure 5 It is a schematic diagram of the capsule endoscope image before deformation provided by the present invention;

[0053] Figure 6 It is a schematic diagram of the capsule endoscope image after deformation provided by the present invention;

[0054] Figure 7 It is a flowchart of the Q-Learning algorithm provided by the present invention;

[0055] Figure 8 It is the third schematic diagram of the process of the method for classifying WCE image scenes based on reinforcement learning provided by the present invention;

[0056] Figure 9 It is a schematic structural diagram of the device for classifying WCE image scenes based on reinforcement learning provided by the present invention;

[0057] Figure 10 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0058] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0059] Figure 1 It is the first schematic diagram of the process of the method for classifying WCE image scenes based on reinforcement learning provided by the present invention, and specifically relates to the scene classification processing of capsule endoscope image data based on a capsule endoscope device. The method for classifying WCE image scenes based on reinforcement learning includes:

[0060] Step 101: Use a preset convolutional neural network model to extract features from each current image frame in all current capsule endoscope images, obtain a feature sequence for each current image frame, and input all the current image frame feature sequences into a preset recurrent neural network model to obtain a first probability prediction result for each current image frame output by the preset recurrent neural network model;

[0061] Step 102: Determine all current image frames with the first probability prediction result less than a preset threshold as secondary classification image frames. For any secondary classification image frame, perform image processing on the secondary classification image frame to obtain a processed image frame, use the Q-Learning algorithm to select a target action, and perform image enhancement on the processed image frame according to the target action to obtain an enhanced image frame;

[0062] Step 103: Input the enhanced image frames into the preset recurrent neural network model to obtain the second probability prediction results of each enhanced image frame output by the preset recurrent neural network model;

[0063] Step 104: Determine the target prediction results of each current image frame in all current capsule endoscopy images according to the first probability prediction results and the second probability prediction results;

[0064] The preset recurrent neural network model is determined by inputting a sample image frame feature sequence into an initial recurrent neural network model to obtain sample probability prediction results, sampling sample actions using a polynomial distribution, continuously updating the first Q value using the sample actions and states, obtaining the maximum expected cumulative return and cross-entropy loss using the sample action corresponding to the maximum first Q value and the reward value, and updating the model parameters corresponding to the initial recurrent neural network model.

[0065] Before step 101, before using the preset convolutional neural network model to extract features from each current image frame in all current capsule endoscopy images, the method further includes:

[0066] Use a capsule endoscope to obtain the original capsule endoscopy images;

[0067] Perform stretching and compression processing on the edges of each original capsule endoscopy image to obtain all current capsule endoscopy images.

[0068] In an optional embodiment, there are 50,000 to 80,000 frames of images in a case of capsule endoscopy images. An original capsule endoscopy image is as Figure 5 shown. It includes an obvious black background and a circular boundary around it, but these areas do not contain valid information. When directly extracting features from the original image, the black background and the edges will cause visual pollution and affect the subsequent task implementation. Therefore, the present invention introduces a geometric transformation method based on spatial coordinate mapping. This method adjusts the spatial distribution of the image pixels to highlight the features in the central area of the image and compress or remove the redundant areas in the periphery at the same time. The mathematical expression of the deformation is as follows:

[0069]

[0070] where x and y are the positions of the image pixels in the original coordinate system; u and v are the new positions of the pixels in the deformed coordinate system. Through this formula, the stretching and compression of the image edges are performed, so that the information density in the central area increases, while the edge area is gradually weakened. After the deformation processing, as Figure 6 shown, the redundant background and circular boundary of the WCE image frames are removed, and the maximum amount of information is retained.

[0071] In step 101, a preset convolutional neural network model is used to extract features from each current image frame in all current capsule endoscopy images, obtaining a feature sequence for each current image frame. Specifically, for the data after deformation processing, taking each case as a unit, it needs to go through the preset convolutional neural network model CNN for feature extraction. CNN extracts local spatial features from the input single-frame image, including information such as texture, edge color distribution, and shape. These information are crucial for the scene classification of WCE image frames. Through convolutional operations, CNN can capture local features at different scales, thus supporting subsequent temporal analysis and classification. Here, the CNN used is the ResNet50 network, and the weights pre-trained on the ImageNet dataset are loaded.

[0072] ResNet50 is a residual network with 50 layers in depth. It performs excellently in multiple computer vision tasks. ResNet50 introduces residual connections. By learning the "residual mapping" rather than the direct mapping, it makes the network easier to optimize. The depth of ResNet50 is moderate, which can achieve a good balance between feature extraction ability and computational cost, and is very suitable for real-time analysis scenarios such as capsule endoscopy videos. By loading the weights pre-trained on the ImageNet dataset, the model convergence time can be significantly shortened, while avoiding the high computational cost of training from scratch.

[0073] After completing the CNN feature extraction, the system stores the features of each WCE case in the HDF5 format, and at the same time records the patient name, total number of frames, and label information of this case. The set Y expression of the labels is as follows, where C is the number of classification categories (the classification categories are stomach, duodenum, jejunum, ileum, large intestine), and y t represents the t-th frame in the set Y, and T is the total number of frames:

[0074] Y = {y t |y t ∈ {0, 1, …, C}, t = 1, … T};

[0075] This structured storage method lays the foundation for subsequent analysis and classification tasks.

[0076] Optionally, the using of the preset convolutional neural network model to extract features from all sample image frames, obtaining a sample image frame feature sequence, includes:

[0077] Using the preset convolutional neural network model to extract features from all sample image frames, obtaining sample extraction features;

[0078] For each sample image frame, it is stored in the HDF5 format, and the patient name, total number of frames, and sample label result corresponding to the features extracted from the sample are recorded;

[0079] Among them, the obtained feature sequence is input into the BiRNN network according to the case. The WCE image sequence of each case is represented as the feature sequence of the sample image frame. The feature sequence of the sample image frame is V = {X1, X2, …, X T}, where X t represents the feature vector of the t-th frame, T is the total number of frames, and V represents the case.

[0080] Optionally, the preset recurrent neural network model BiRNN network captures the temporal dependencies between frames through a recursive structure:

[0081] h t = f(W h ·h t-1 + W x ·x t + b h );

[0082] Among them, h t represents the hidden state of the t-th frame, W h , W x are weight matrices, b h is the bias term, and f(·) is the activation function. The bidirectional recurrent neural network combines forward and backward time information. As shown in the following formula, it captures context information by concatenating the forward hidden state and the backward hidden state .

[0083]

[0084] The classification probability of each frame is calculated through the Softmax layer. As shown in the following formula, where P(x t ) is the classification probability distribution of the t-th frame (the classification categories are stomach, duodenum, jejunum, ileum, large intestine), W i and b o are the weights and biases of the output layer respectively, is the hidden state of the t-th frame under the bidirectional bi.

[0085]

[0086] The research of the present invention supports various recurrent neural network structures, such as bidirectional long short-term memory network (BiLSTM), bidirectional gated recurrent unit (BiGRU), Transformer encoder, etc.

[0087] Optionally, before inputting all current image frame feature sequences into a preset recurrent neural network model, the method further includes:

[0088] Using a preset convolutional neural network model to extract features from all sample image frames, obtaining a sample image frame feature sequence, inputting the sample image frame feature sequence into an initial recurrent neural network model, and obtaining a sample probability prediction result;

[0089] Sampling sample actions using a multinomial distribution, obtaining a sample result corresponding to the sample action, obtaining a current reward value using the sample label result and the sample result corresponding to the sample action, determining a current state according to the current reward value and the reward value before sampling, and continuously updating the first Q value using the current state;

[0090] Determining an expected cumulative return using the action corresponding to the maximum first Q value and the reward value, calculating a gradient of the expected cumulative return, updating model parameters corresponding to the initial recurrent neural network model using a preset formula, obtaining target model parameters, and determining the preset recurrent neural network model according to the target model parameters.

[0091] In this optional embodiment, the present invention constructs a hybrid reinforcement learning framework to optimize the BiRNN network parameters. Reinforcement learning (RL) is an important branch of machine learning. By interacting with the environment through an agent, it learns the best strategy to maximize the long-term reward. There are various methods for reinforcement learning. The present invention combines the advantages of the Monte Carlo policy gradient (REINFORCE) and the action-value function (Q-Learning) algorithms, and simultaneously optimizes the policy distribution to improve the exploration ability and sample efficiency of the agent.

[0092] In each round of training, the calculation is performed case by case according to the number of cases, and the training process for each case is the same. First, obtain the probability distribution P(x t ) of each frame in each case obtained from the activation function of the BiRNN output. Sample an action a from the probability distribution. The policy used for action sampling is the multinomial distribution, and the formula is as follows:

[0093] a t ~CategoricalP(x t ));

[0094] Sample according to the probability distribution of each frame to obtain the predicted category of this frame, and thus obtain a sequence A with predicted categories. The expression is as shown in the following formula, where a t represents the sampling of the t-th frame:

[0095] A = {a t |a t∈ {0, 1, ..., C}, et = 1, ..., T};

[0096] The reward value is obtained by comparing the sampled set A with the label Y. As shown in the following formula, Vi is the i-th case, and is the reward for the i-th case.

[0097]

[0098] Construct the first Q-value storage table, initialize the first Q-value to 0, set the vertical axis of the Q-table as the mean of the sampled action probabilities, and the horizontal axis as the state. Set the state to two, and the current state is determined after sampling. Let R0 be the initial reward value when the probability of the RNN output is not sampled, and the reward after sampling is R1. The new state is determined according to whether R1 is less than, equal to, or greater than R0 respectively. When R1 ≥ R0, it is state s1, and when R1 < R0, it is s2. a n is the n-th action. The initial Q-table is as follows:

[0099] Table 1 Initial Q-table

[0100] Status / Action <![CDATA[a1]]> <![CDATA[a2]]> a… <![CDATA[a n > <![CDATA[s1(R1≥R0)]]> 0 0 … 0 <![CDATA[s2(R1<R0)]]> 0 0 … 0

[0101] The update of the first Q-value in Table 1 is based on the Bellman equation, and the equation formula is as follows:

[0102]

[0103] where α is the learning rate, θ is the discount factor, s is the state, a is the action, R is the reward value, is the maximum first Q-value of the next state. After looping N times, select the action corresponding to the maximum first Q-value in the Q-table and its corresponding reward value. The flowchart of the Q-Learning algorithm is as Figure 7 shown.

[0104] The goal of the above embodiment is to maximize the expected cumulative return of the policy:

[0105]

[0106] where p θ (a 1:T ) represents the probability distribution of the action corresponding to the maximum first Q-value after sampling. T is the total number of frames of this case, and the calculation of R can refer to the above calculation formula, π θ is defined by the network trained based on hybrid reinforcement learning. To maximize J(θ), it is necessary to take the gradient of the policy parameter θ. The formula for the policy gradient is:

[0107]

[0108] Among them, a t is the action taken at time t, and h t is the hidden state from the RNN.

[0109] Meanwhile, the cross-entropy loss function (L CE ) is introduced as one of the loss terms for model optimization to measure the difference between the predicted distribution and the true distribution. The formula is as follows:

[0110]

[0111] Among them, T is the total number of frames, C is the number of classification categories, y t is the label of the t-th frame, and a t,i is the action sampling of the t-th frame as the i-th category.

[0112] In addition, an l2 regularization term is added to the weight parameter θ to avoid overfitting. The formula is as follows:

[0113]

[0114] Among them, θ i,j represents the weight value of a certain layer or a certain connection in the model, and the indices i and j represent the positions in the weight matrix.

[0115] Combining the above formulas, updating the model parameters corresponding to the initial recurrent neural network model using the preset formula to obtain the target model parameters includes:

[0116]

[0117] Among them, α is the learning rate, β1 and β2 are hyperparameters for balancing weights, L CE is the cross-entropy loss function, L weight is the regularization term, and J is the expected cumulative return mentioned above (the formula is represents the overall derivative of the parameter θ in (-J + β1L CE + β2L weight ) to obtain the gradient.

[0118] Through the above method, continuously update θ, and use Adam as the optimizer to train the preset recurrent neural network model based on reinforcement learning for the prediction of WCE sequence frames, so that in step 101, all current image frame feature sequences can be input into the preset recurrent neural network model to obtain the first probability prediction result of each current image frame output by the preset recurrent neural network model.

[0119] In step 102, the first probability prediction result is the predicted probability of different scene classifications, and the scene classifications include the stomach, duodenum, jejunum, ileum, and large intestine;

[0120] Determining all current image frames with the first probability prediction result less than a preset threshold includes:

[0121] For any current image frame, when the prediction probability of each scene classification in the first probability prediction result is less than the preset threshold, determining the current image frame as a secondary classification image frame;

[0122] Traverse all current image frames until all secondary classification image frames are determined.

[0123] Optionally, the present invention preliminarily screens the input test data, extracts its features and determines whether they meet the requirements. First, input the test case image sequence V = {X1, X2, …, X T}, where X t represents the t-th frame image. Input each frame image X t into the CNN network in the first stage to extract features, and the extracted features are represented as:

[0124]

[0125] where F i represents the feature vector corresponding to the i-th frame image, and the dimension is d. Input the extracted features into the model trained in the first stage, predict the result and obtain the classification probability distribution of each frame. The probability values of each category of each frame image output by the model. By setting a probability threshold λ, here λ is taken as 0.6.

[0126] If the prediction probability of a certain category is greater than or equal to the threshold, it is considered that the image has been correctly classified. If the probability is lower than the threshold, it is considered that the image classification is incorrect and needs further processing. Based on this judgment, the images are divided into two categories, the determined classified images D pass (higher than the threshold) and the images with uncertain classification D opt (lower than the threshold).

[0127] Calculate the feature standard deviation M i for the images with uncertain frames, and the formula is as follows:

[0128]

[0129] where i represents the i-th frame, F i,j is the j-th component of the feature vector F in the i-th frame, μ i is the mean value of F i , and the calculation formula is:

[0130]

[0131] Optionally, the image processing of the secondary classification image frames includes rotating the image and scaling the image;

[0132] Using the Q-Learning algorithm to select a target action, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame, including:

[0133] For any processed image frame, using a preset convolutional neural network model to extract features from the processed image frame to obtain processed image features, and calculating the first standard deviation of the current processed image frame;

[0134] Calculating the second standard deviation of the secondary classification image frame corresponding to the current processed image frame, and determining a target reward value according to the first standard deviation and the second standard deviation;

[0135] Updating the second Q value according to the target reward value, and using the action corresponding to the maximum second Q value to process the secondary classification image frame to obtain an enhanced image frame;

[0136] Traverse all processed image frames to obtain all enhanced image frames.

[0137] After initially screening out the image frames with uncertain predictions, in order to improve the quality of these image frames, a multi-view feature transformation method is adopted, combined with the Q-Learning algorithm of reinforcement learning to find the optimal action and perform image enhancement on the data. The operations performed on these image frames are rotating the image and scaling the image. In this part, i represents the i-th frame. The rotation of the image is:

[0138]

[0139] where R(X i , θ) represents the transformation function for rotating the i-th frame image X by an angle θ. The scaling of the image is:

[0140]

[0141] where S(X i , s) represents the transformation function for scaling the i-th frame image X by a scale factor s. The generated transformation data set is Input each transformed image X i 变换 into the CNN in the first stage to extract the corresponding feature vector F i 变换 , and then use the calculation formula for calculating the feature standard deviation M i of the uncertain frame image to calculate its feature standard deviation M i 变换 . Combining reinforcement learning, find the optimal action a for each input imagei , select the final transformation for secondary prediction. The design state has two states. When the standard deviation M1 of the features after the transformation of this frame is greater than or equal to the standard deviation M of the features before the transformation, it is in state S1; otherwise, it is in state S2. The reward r is also designed based on the standard deviation of the features, as follows:

[0142]

[0143] When M1 is greater than M, the reward value is incremented by one; when M1 is equal to M, the reward value is zero; when M1 is less than M, the reward value is decremented by one.

[0144] The number of iterations of the Q-table in Q-Learning is N = a × m, where a is the number of actions, which is 3, and m is a constant. After each iteration, the Q-table contains the state and action, denoted as Q(s,a). The Q-Learning update rule follows the Bellman equation formula. Select the action corresponding to the second largest Q value in the Q-table Generate an optimized image The optimized data set is denoted as

[0145] In step 103, the optimized data set in step 102 will be input into a pre-trained convolutional neural network (CNN) for feature extraction. The extracted features are further input into the pre-trained preset recurrent neural network model in step 101 above, and the second probability prediction result of each enhanced image frame output by the preset recurrent neural network model is obtained.

[0146] In step 104, determining the target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result includes:

[0147] For any enhanced image frame, when the probability prediction result is greater than the preset threshold, determine the second probability prediction result corresponding to the enhanced image frame as the final prediction result of the enhanced image frame; otherwise, determine the first probability prediction result corresponding to the enhanced image frame as the final prediction result of the enhanced image frame;

[0148] Traverse all enhanced image frames, determine the final prediction result corresponding to each enhanced image frame, determine all current image frames with the first probability prediction result greater than or equal to the preset threshold as unenhanced image frames, and determine the first probability prediction result corresponding to the unenhanced image frames as the final prediction result of the unenhanced image frames;

[0149] Determine the target prediction result of each current image frame in all current capsule endoscopy images according to the final prediction result of the enhanced image frame and the final prediction result of the unenhanced image frame.

[0150] Optionally, if the prediction probability of the enhanced image frame exceeds a set threshold, it is considered that the classification result is reliable enough, and this result is used as the final result of this frame. Otherwise, the first prediction result is used as the final result. Finally, the prediction result of the optimized data is combined with the classified data D determined in step 101 pass to perform result merging to form a final classification result.

[0151] The present invention can reduce the reading time of WCE, relieve the work burden, and accurately locate the target scene by automatically classifying the stomach, small intestine, and large intestine, which is of great significance to the image automatic detection algorithm based on deep learning and helps to improve the detection accuracy and efficiency. The present invention focuses on the classification of scenes such as the stomach, duodenum, jejunum, ileum, and large intestine in the capsule endoscopy image sequence, and proposes a framework combining deep learning and hybrid reinforcement learning for efficient and accurate classification of various scenes of capsule endoscopy images. A hybrid reinforcement learning is used, and two reinforcement learning methods, Monte Carlo policy gradient (REINFORCE) and action-value function (Q-Learning), are combined to classify the WCE image scene, rather than just using a single reinforcement learning method. This not only makes up for the deficiencies of a single reinforcement learning method and improves the classification accuracy, but also evaluates the credibility of the classification result through a probability threshold method in the stage of automatically adjusting the image enhancement to correct the test result, and automatically distinguishes the images that have been correctly classified from the images with uncertain classification results. For the latter, image enhancement operations will be performed to eliminate potential classification errors. The image enhancement adjusts the enhancement strategy by estimating the action value to ensure that the image quality is optimized. The present invention not only optimizes the performance of the classification model using hybrid reinforcement learning, but also uses secondary prediction to improve the processing ability of uncertain classification results, and has strong practical application value.

[0152] Figure 2It is the second flow schematic diagram of the WCE image scene classification method based on reinforcement learning provided by the present invention. For the capsule endoscopy image frames, in the first stage of training the WCE image scene classification network based on reinforcement learning, deformation preprocessing is performed on all case WCE image frames. The CNN network extracts the features of the WCE image frames, the BiRNN network predicts the probability of each frame, and the Q-Learning algorithm and policy gradient are combined to optimize the training model parameters. In the second stage of automatically adjusting the image enhancement to correct the test results, the trained model is used to test each frame of the image, and it is determined whether the classification result is credible. For the images with uncertain classification, the Q-Learning algorithm is used to select the optimal action, and the optimal action is used for image enhancement. The enhanced image is sent into the model for prediction again, and the first prediction result and the second prediction result are combined to obtain the final result, and the final classification result is output.

[0153] Figure 3 It is the training network structure diagram provided by the present invention. This training method adopts an innovative end-to-end framework, which organically combines the feature extraction network and temporal network in deep learning with the reinforcement learning method, fully explores the potential of spatio-temporal features and policy optimization, and provides an effective solution for the WCE image frame classification task. The training network structure is as Figure 3 shown. First, the input image sequence V i is processed by a Convolutional Neural Network (CNN) to extract the features of the image. Then, a Bidirectional Recurrent Neural Network (BiRNN) is used to process the temporal dependence between frames, capture the dynamic information and temporal features in the video data, and output the prediction probability p t of each frame. The reinforcement learning module makes action selection for the category of the frame based on the polynomial distribution and calculates the corresponding reward, and calculates the Q-value function through the Q-Learning algorithm to select the optimal action. The selected optimal action is used to update the policy and loss function. By jointly optimizing the objective functions of supervised learning and reinforcement learning, the overall performance of the WCE image frame scene classification is improved.

[0154] Figure 4 It is the test network structure diagram provided by the present invention. When loading the trained model for classification prediction, in the case of inaccurate classification, it is necessary to perform image enhancement on the misclassified image frames to improve the final classification performance. The process is as Figure 4As shown below. First, use the trained model to predict the test image sequence, and according to a preset probability threshold, divide the prediction results into images with definite classification (i.e., images with prediction probability exceeding the threshold) and images with uncertain classification (i.e., images with prediction probability lower than the threshold). For image frames with uncertain classification, use the Q-Learning method to select the optimal image enhancement strategy. The enhancement strategies include rotating the image by 90°, rotating the image by 180°, and scaling the image to 75%. Through multiple rounds of Q-value calculation, select the action corresponding to the maximum Q-value, and enhance the image according to this action. The enhanced image will then be input into the model for secondary classification to obtain the prediction results of the second round. If the prediction probability of the enhanced image exceeds the set threshold, it is considered that the classification of the enhanced image is reliable enough and can be used as the final classification result of this frame; otherwise, retain the original classification result. The final classification result is generated by synthesizing the prediction results of the first and second classifications, thereby further optimizing the overall classification performance on the basis of ensuring the accuracy of the original test image classification result.

[0155] Figure 8 FIG. 3 is the third schematic flowchart of the WCE image scene classification method based on reinforcement learning provided by the present invention. For the wireless capsule endoscopy image scene classification based on reinforcement learning, a method combining deep learning and reinforcement learning is used to realize the scene classification of wireless capsule endoscopy images. The specific process is as Figure 8 shown, and the specific process is as follows:

[0156] Step 1: Input the capsule endoscopy image sequence frames;

[0157] Step 2: Deformation processing of the capsule endoscopy images;

[0158] Step 3: Use the pre-trained CNN network to extract features from the image frames of each case, and save the features in the HDF5 format, including relevant information of the case: patient name, total number of frames, classification label, etc.;

[0159] Step 4: Input the feature sequence of the image frames of the case into a bidirectional recurrent neural network (BiRNN), and the BiRNN network outputs the prediction probability distribution of each frame;

[0160] Step 5: Use the multinomial distribution to perform action sampling on the classification probability distribution of each frame output by the BiRNN, and design reasonable rewards and states, and use the Q-Learning algorithm to select the rewards and sampling actions under the maximum Q-value in N rounds of sampling to calculate the policy gradient and cross-entropy loss function, and finally train the model by optimizing the model parameters θ;

[0161] Step 6: Input the test image sequence into the model trained in Step 5 for prediction, and divide it into a set of images with definite prediction classification and a set of images with uncertain classification;

[0162] Step 7: Use the Q-Learning algorithm to select the optimal action for the set of images with uncertain classification. Select the optimal action to perform image enhancement on the images, input the enhanced images into the trained model to predict the results again, and compare the prediction results with the threshold. If the predicted probability exceeds the set threshold, use the second prediction result as the final classification result for this frame; otherwise, retain the first classification result. Obtain the final prediction results for the images with uncertain classification.

[0163] Step 8: Combine the results of the classification determined in Step 6 and the results predicted in Step 7 to obtain the final result.

[0164] Figure 9 FIG. 9 is a schematic structural diagram of a WCE image scene classification device based on reinforcement learning provided by the present invention. The WCE image scene classification device based on reinforcement learning includes an input unit 1. The input unit 1 is configured to extract features from each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model, obtain a feature sequence for each current image frame, and input all the feature sequences of the current image frames into a preset recurrent neural network model to obtain a first probability prediction result for each current image frame output by the preset recurrent neural network model. The working principle of the input unit 1 can refer to the foregoing step 101 and will not be elaborated herein.

[0165] The WCE image scene classification device based on reinforcement learning further includes an image processing unit 2. The image processing unit 2 is configured to determine all current image frames with a first probability prediction result less than a preset threshold as secondary classification image frames. For any secondary classification image frame, perform image processing on the secondary classification image frame to obtain a processed image frame, select a target action by using the Q-Learning algorithm, and perform image enhancement on the processed image frame according to the target action to obtain an enhanced image frame. The working principle of the determination unit 2 can refer to the foregoing step 102 and will not be elaborated herein.

[0166] The WCE image scene classification device based on reinforcement learning further includes an output unit 3. The output unit 3 is configured to input the enhanced image frame into the preset recurrent neural network model to obtain a second probability prediction result for each enhanced image frame output by the preset recurrent neural network model. The working principle of the output unit 3 can refer to the foregoing step 103 and will not be elaborated herein.

[0167] The WCE image scene classification device based on reinforcement learning further includes a determination unit 4, which is configured to determine the target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result. The working principle of the determination unit 4 can refer to the foregoing step 104 and will not be elaborated here.

[0168] The preset recurrent neural network model is determined by inputting a feature sequence of a sample image frame into an initial recurrent neural network model to obtain a sample probability prediction result, sampling a sample action using a multinomial distribution, continuously updating a first Q value using the sample action and state, obtaining a maximum expected cumulative return and a cross-entropy loss using the sample action corresponding to the maximum first Q value and a reward value, and updating the model parameters corresponding to the initial recurrent neural network model.

[0169] Figure 10 is a schematic structural diagram of an electronic device provided by the present invention. As Figure 10As shown, the electronic device may include: a processor 110, a communications interface 120, a memory 130, and a communication bus 140. Among them, the processor 110, the communications interface 120, and the memory 130 complete communication with each other through the communication bus 140. The processor 110 may call logical instructions in the memory 130 to execute a WCE image scene classification method based on reinforcement learning. The method includes: extracting features from each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model to obtain a feature sequence for each current image frame, inputting all the current image frame feature sequences into a preset recurrent neural network model to obtain a first probability prediction result for each current image frame output by the preset recurrent neural network model; determining all current image frames with the first probability prediction result less than a preset threshold as secondary classification image frames. For any secondary classification image frame, performing image processing on the secondary classification image frame to obtain a processed image frame, selecting a target action by using the Q-Learning algorithm, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame; inputting the enhanced image frame into the preset recurrent neural network model to obtain a second probability prediction result for each enhanced image frame output by the preset recurrent neural network model; determining a target prediction result for each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result; the preset recurrent neural network model is determined by inputting a sample image frame feature sequence into an initial recurrent neural network model to obtain a sample probability prediction result, sampling sample actions by using a polynomial distribution, continuously updating a first Q value by using the sample actions and states, obtaining a maximum expected cumulative return and a cross-entropy loss by using the sample action corresponding to the maximum first Q value and a reward value, and updating model parameters corresponding to the initial recurrent neural network model.

[0170] In addition, the logical instructions in the aforementioned memory 130 can be implemented in the form of software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, external hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0171] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a method for classifying WCE image scenes based on reinforcement learning provided by each of the above methods. The method includes: extracting features from each current image frame in all current capsule endoscopy images using a preset convolutional neural network model to obtain a feature sequence for each current image frame; inputting all the current image frame feature sequences into a preset recurrent neural network model to obtain a first probability prediction result for each current image frame output by the preset recurrent neural network model; determining all current image frames with a first probability prediction result less than a preset threshold as secondary classification image frames. For any secondary classification image frame, performing image processing on the secondary classification image frame to obtain a processed image frame, selecting a target action using the Q-Learning algorithm, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame; inputting the enhanced image frame into the preset recurrent neural network model to obtain a second probability prediction result for each enhanced image frame output by the preset recurrent neural network model; determining a target prediction result for each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result; the preset recurrent neural network model is determined by inputting a sample image frame feature sequence into an initial recurrent neural network model to obtain a sample probability prediction result, sampling sample actions using a multinomial distribution, continuously updating the first Q value using the sample actions and states, obtaining the maximum expected cumulative return and cross-entropy loss using the sample action corresponding to the maximum first Q value and the reward value, and updating the model parameters corresponding to the initial recurrent neural network model.

[0172] In another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements a WCE image scene classification method based on reinforcement learning provided by the above-mentioned various methods. The method includes: extracting features from each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model to obtain a feature sequence of each current image frame, inputting the feature sequences of all current image frames into a preset recurrent neural network model to obtain a first probability prediction result of each current image frame output by the preset recurrent neural network model; determining all current image frames with the first probability prediction result less than a preset threshold as secondary classification image frames, for any secondary classification image frame, performing image processing on the secondary classification image frame to obtain a processed image frame, selecting a target action by using the Q-Learning algorithm, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame; inputting the enhanced image frame into the preset recurrent neural network model to obtain a second probability prediction result of each enhanced image frame output by the preset recurrent neural network model; determining a target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result; the preset recurrent neural network model is determined by inputting a feature sequence of a sample image frame into an initial recurrent neural network model to obtain a sample probability prediction result, sampling a sample action by using a polynomial distribution, continuously updating a first Q value by using the sample action and state, obtaining a maximum expected cumulative return and cross-entropy loss by using the sample action corresponding to the maximum first Q value and a reward value, and updating model parameters corresponding to the initial recurrent neural network model.

[0173] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0174] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for classifying WCE image scenes based on reinforcement learning, characterized in that, Including: Performing feature extraction on each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model to obtain a feature sequence of each current image frame, and inputting the feature sequences of all current image frames into a preset recurrent neural network model to obtain a first probability prediction result of each current image frame output by the preset recurrent neural network model; Determining all current image frames with the first probability prediction result less than a preset threshold as secondary classification image frames. For any secondary classification image frame, performing image processing on the secondary classification image frame to obtain a processed image frame, selecting a target action by using the Q-Learning algorithm, and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame; Inputting the enhanced image frame into the preset recurrent neural network model to obtain a second probability prediction result of each enhanced image frame output by the preset recurrent neural network model; Determining a target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result; The preset recurrent neural network model is determined by inputting a feature sequence of a sample image frame into an initial recurrent neural network model to obtain a sample probability prediction result, sampling a sample action by using a polynomial distribution, continuously updating a first Q value by using the sample action and a state, obtaining a maximum expected cumulative return and a cross-entropy loss by using the sample action corresponding to the maximum first Q value and a reward value, and updating model parameters corresponding to the initial recurrent neural network model.

2. The method for classifying WCE image scenes based on reinforcement learning according to claim 1, wherein Before performing feature extraction on each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model, the method further includes: Obtaining an original capsule endoscopy image by using a capsule endoscope; Performing stretching and compression processing on the edges of each original capsule endoscopy image to obtain all current capsule endoscopy images.

3. The method for classifying WCE image scenes based on reinforcement learning according to claim 1, wherein Before inputting the feature sequences of all current image frames into a preset recurrent neural network model, the method further includes: Performing feature extraction on all sample image frames by using a preset convolutional neural network model to obtain a feature sequence of a sample image frame, and inputting the feature sequence of the sample image frame into an initial recurrent neural network model to obtain a sample probability prediction result; Sampling a sample action by using a polynomial distribution to obtain a sample result corresponding to the sample action, obtaining a current reward value by using a sample label result and the sample result corresponding to the sample action, determining a current state according to the current reward value and a reward value before sampling, and continuously updating a first Q value by using the current state; Determining an expected cumulative return by using the action corresponding to the maximum first Q value and a reward value, calculating a gradient of the expected cumulative return, updating model parameters corresponding to the initial recurrent neural network model by using a preset formula to obtain target model parameters, and determining the preset recurrent neural network model according to the target model parameters.

4. The method for classifying WCE image scenes based on reinforcement learning according to claim 3, characterized in that, The performing feature extraction on all sample image frames by using a preset convolutional neural network model to obtain a feature sequence of a sample image frame includes: Performing feature extraction on all sample image frames by using a preset convolutional neural network model to obtain a sample extraction feature; For each sample image frame, it is stored in the HDF5 format, and the patient name, total number of frames, and sample label result corresponding to the features extracted from the sample are recorded; Among them, the sample image frame feature sequence is V = {X1, X2, …, X T}, where X t represents the feature vector of the t-th frame, T is the total number of frames, and V represents the case.

5. The method for classifying WCE image scenes based on reinforcement learning according to claim 3, wherein, Updating the model parameters corresponding to the initial recurrent neural network model using a preset formula to obtain target model parameters, including: where α is the learning rate, β1 and β2 are hyperparameters for balancing weights, L CE is the cross-entropy loss function, L weight is the regularization term, J is the expected cumulative return, denotes the overall derivative with respect to the parameter θ in (-j + β1L CE + β2L weight ) to obtain the gradient.

6. The method for classifying WCE image scenes based on reinforcement learning according to claim 1, wherein, The first probability prediction result is the prediction probability for different scene classifications, and the scene classifications include the stomach, duodenum, jejunum, ileum, and large intestine; Determining all current image frames with the first probability prediction result less than a preset threshold, including: For any current image frame, when the prediction probability for each scene classification in the first probability prediction result is less than the preset threshold, determining the current image frame as a secondary classification image frame; Traversing all current image frames until all secondary classification image frames are determined.

7. The method for classifying WCE image scenes based on reinforcement learning according to claim 1, characterized in that, Processing the secondary classification image frames includes rotating the image and scaling the image; Selecting a target action using the Q-Learning algorithm and performing image enhancement on the processed image frame according to the target action to obtain an enhanced image frame, including: For any processed image frame, using a preset convolutional neural network model to extract features from the processed image frame to obtain processed image features, and calculating the first standard deviation of the current processed image frame; Calculating the second standard deviation of the secondary classification image frame corresponding to the current processed image frame, and determining a target reward value according to the first standard deviation and the second standard deviation; Updating the second Q value according to the target reward value, and processing the secondary classification image frame using the action corresponding to the maximum second Q value to obtain an enhanced image frame; Traversing all processed image frames to obtain all enhanced image frames.

8. The method for classifying WCE image scenes based on reinforcement learning according to claim 1, wherein Determining the target prediction result of each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result, including: For any enhanced image frame, when the probability prediction result is greater than the preset threshold, determining the second probability prediction result corresponding to the enhanced image frame as the final prediction result of the enhanced image frame; otherwise, determining the first probability prediction result corresponding to the enhanced image frame as the final prediction result of the enhanced image frame; Traversing all enhanced image frames, determining the final prediction result corresponding to each enhanced image frame, determining all current image frames with the first probability prediction result greater than or equal to the preset threshold as unenhanced image frames, and determining the first probability prediction result corresponding to the unenhanced image frame as the final prediction result of the unenhanced image frame; Determining the target prediction result of each current image frame in all current capsule endoscopy images according to the final prediction result of the enhanced image frame and the final prediction result of the unenhanced image frame.

9. An apparatus for classifying WCE image scenes based on reinforcement learning, characterized in that, Including: An input unit, which is configured to extract features from each current image frame in all current capsule endoscopy images by using a preset convolutional neural network model, obtain a feature sequence for each current image frame, and input all the current image frame feature sequences into a preset recurrent neural network model to obtain a first probability prediction result for each current image frame output by the preset recurrent neural network model; An image processing unit, which is configured to determine all current image frames with a first probability prediction result less than a preset threshold as secondary classification image frames. For any secondary classification image frame, perform image processing on the secondary classification image frame to obtain a processed image frame, select a target action by using the Q-Learning algorithm, and perform image enhancement on the processed image frame according to the target action to obtain an enhanced image frame; An output unit, which is configured to input the enhanced image frame into the preset recurrent neural network model to obtain a second probability prediction result for each enhanced image frame output by the preset recurrent neural network model; A determination unit, which is configured to determine a target prediction result for each current image frame in all current capsule endoscopy images according to the first probability prediction result and the second probability prediction result; The preset recurrent neural network model is determined by inputting a sample image frame feature sequence into an initial recurrent neural network model to obtain a sample probability prediction result, sampling sample actions by using a polynomial distribution, continuously updating a first Q value by using the sample actions and states, obtaining a maximum expected cumulative return and a cross-entropy loss by using the sample action corresponding to the maximum first Q value and a reward value, and updating the model parameters corresponding to the initial recurrent neural network model.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the WCE image scene classification method based on reinforcement learning according to any one of claims 1 to 8.

Images