Capsule endoscopy path prediction method and device based on multimodal data fusion

Abstract

The present invention provides a capsule endoscope path prediction method and device based on multimodal data fusion, which belongs to the field of medical equipment technology, wherein the method comprises: collecting patient's body position time series data in the current time period through multiple inertial measurement units IMU, collecting patient's digestive tract time series image data in the current time period through capsule endoscope; preprocessing body position time series data and time series image data to obtain preprocessed body position time series data and preprocessed time series image data; inputting preprocessed body position time series data and preprocessed time series image data into a pre-trained prediction model, and obtaining capsule endoscope path prediction results and patient's body position adjustment prompts output by the prediction model. The present invention improves the accuracy of capsule endoscope path prediction and can effectively avoid missed detection.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a capsule endoscope path prediction method and device based on multimodal data fusion. Background Art

[0002] At present, capsule endoscopy technology is widely used in gastrointestinal examinations. Unpowered capsule endoscopy uses the natural peristalsis of the digestive tract and gravity to push the capsule forward. However, due to the uncontrollability of the path of unpowered capsule endoscopy, there is a high risk of missed detection in some areas, especially the complex stomach area. How to accurately predict the path of the capsule endoscopy is a key issue that needs to be solved urgently. At present, the existing technology relies on a single data source to predict the path of the capsule endoscopy, which affects the accuracy of the path prediction and easily causes the risk of missed detection. Summary of the invention

[0003] The present invention provides a capsule endoscope path prediction method and device based on multimodal data fusion, so as to solve the defect that the path of the capsule endoscope is predicted by relying on a single data source in the prior art, which affects the accuracy of the path prediction and easily causes the risk of missed detection.

[0004] The present invention provides a capsule endoscopy path prediction method based on multimodal data fusion, comprising:

[0005] Collecting time-series data of the patient's body position in a current time period by using a plurality of inertial measurement units (IMUs), and collecting time-series image data of the patient's digestive tract in a current time period by using a capsule endoscope;

[0006] Preprocessing the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data;

[0007] Inputting the preprocessed body position time series data and the preprocessed time series image data into a pre-trained prediction model, and obtaining the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the prediction model;

[0008] The prediction model is obtained by training based on preprocessed sample body position time series data and preprocessed sample time series image data, as well as sample capsule endoscope path labels and sample patient body position adjustment prompt labels.

[0009] In some embodiments, the prediction model includes a body position feature extraction layer, an image feature extraction layer, a feature fusion layer, and an output layer;

[0010] Correspondingly, the preprocessed posture time series data and the preprocessed time series image data are input into a pre-trained prediction model to obtain the capsule endoscope path prediction result and the patient's posture adjustment prompt output by the prediction model, including:

[0011] Inputting the preprocessed body position time series data into the body position feature extraction layer to obtain the time series features output by the body position feature extraction layer;

[0012] Inputting the preprocessed time-series image data into the image feature extraction layer to obtain image features output by the image feature extraction layer;

[0013] Inputting the time series features and the image features into the feature fusion layer to obtain fusion features output by the feature fusion layer;

[0014] The fusion feature is input into the output layer, and the path prediction result of the capsule endoscope and the body position adjustment prompt of the patient are obtained as output by the output layer.

[0015] In some embodiments, the body position feature extraction layer includes a Transformer layer based on a multi-head self-attention mechanism, the image feature extraction layer includes a lightweight convolutional neural network CNN layer, and the image features include refined regional classification results of the preprocessed time series image data.

[0016] In some embodiments, the method further comprises:

[0017] After the patient adjusts his / her body position, the changes in the patient's body position and the image data collected by the capsule endoscope are continuously monitored to obtain monitoring data, and a new body position adjustment prompt is generated based on the monitoring data.

[0018] In some embodiments, after the patient adjusts the body position, continuously monitoring the patient's body position change and the image data collected by the capsule endoscope to obtain monitoring data, and generating a new body position adjustment prompt according to the monitoring data, including:

[0019] After the patient has adjusted the body position, the body position monitoring data of the patient is obtained, and according to the body position monitoring data, it is determined whether the patient has completed the body position adjustment according to the body position adjustment prompt, and if so, it is confirmed that the body position adjustment is completed, otherwise, a new body position adjustment prompt is generated to guide the patient to adjust the body position again, and the cycle is repeated until it is confirmed that the body position adjustment is completed;

[0020] After confirming that the body position adjustment is completed, the image data of the digestive tract collected by the capsule endoscope is obtained, and it is determined whether the target area is photographed based on the image data. If so, the instruction is stopped; otherwise, a body position adjustment suggestion is generated so that the capsule endoscope can photograph the target area.

[0021] In some embodiments, the method further comprises:

[0022] Acquire the physiological information of the patient, and adaptively adjust the patient's posture adjustment prompt according to the patient's physiological information.

[0023] In some embodiments, the preprocessing of the body position time series data and the time series image data includes:

[0024] Performing noise filtering and normalization preprocessing on the body position time series data, and performing standardization and data enhancement preprocessing on the time series image data;

[0025] The body position time series data and the time series image data are aligned by using a dynamic time warping (DTW) algorithm.

[0026] In some embodiments, the process of determining the prediction model includes:

[0027] Collecting sample body position time-series data of a sample patient within a sample time period through a plurality of IMUs, and collecting sample time-series image data of the digestive tract of the sample patient within a sample time period through a sample capsule endoscope;

[0028] Preprocessing the sample body position time series data and the sample time series image data to obtain preprocessed sample body position time series data and preprocessed sample time series image data;

[0029] Determining a path label of the sample capsule endoscope and a body position adjustment prompt label of the sample patient;

[0030] The preprocessed sample posture time series data and the preprocessed sample time series image data are used as training data, and the path label of the sample capsule endoscope and the posture adjustment prompt label of the sample patient are used as sample labels to train an initial prediction model. After the training is completed, the prediction model is obtained.

[0031] In some embodiments, the training of the initial prediction model includes:

[0032] Inputting the preprocessed sample body position time series data and the preprocessed sample time series image data into an initial prediction model, and obtaining the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient output by the initial prediction model;

[0033] Calculating a loss function value based on the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient, as well as the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient;

[0034] Based on the loss function value, the parameters of the initial prediction model are iteratively optimized to obtain the prediction model.

[0035] The present invention also provides a capsule endoscope path prediction device based on multimodal data fusion, comprising:

[0036] A data acquisition unit, used to acquire time-series data of the patient's body position in a current time period through a plurality of inertial measurement units (IMUs), and to acquire time-series image data of the patient's digestive tract in the current time period through a capsule endoscope;

[0037] A preprocessing unit, used for preprocessing the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data;

[0038] A prediction unit, used for inputting the preprocessed posture time series data and the preprocessed time series image data into a pre-trained prediction model, and obtaining the capsule endoscope path prediction result and the patient's posture adjustment prompt output by the prediction model;

[0039] The prediction model is obtained by training based on preprocessed sample body position time series data and preprocessed sample time series image data, as well as sample capsule endoscope path labels and sample patient body position adjustment prompt labels.

[0040] The capsule endoscope path prediction method and device based on multimodal data fusion provided by the present invention collect the patient's body position time series data within the current time period through multiple inertial measurement units (IMUs), and collect the patient's digestive tract time series image data within the current time period through a capsule endoscope; preprocess the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data; input the preprocessed body position time series data and the preprocessed time series image data into a pre-trained prediction model to obtain the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the prediction model, thereby improving the accuracy of capsule endoscope path prediction and effectively avoiding missed detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is one of the flow charts of the capsule endoscopy path prediction method based on multimodal data fusion provided in an embodiment of the present invention;

[0043] Figure 2 This is the second flow chart of the capsule endoscopy path prediction method based on multimodal data fusion provided in an embodiment of the present invention;

[0044] Figure 3 It is a flow chart of inputting preprocessed posture time series data and preprocessed time series image data into a pre-trained prediction model provided by an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of a process for continuously monitoring changes in the patient's body position and image data collected by a capsule endoscope provided by an embodiment of the present invention;

[0046] Figure 5 is a flowchart of a process for determining a prediction model provided by an embodiment of the present invention;

[0047] Figure 6 It is a structural schematic diagram of a capsule endoscopy path prediction device based on multimodal data fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Figure 1 This is a flow chart of a capsule endoscopy path prediction method based on multimodal data fusion provided by an embodiment of the present invention. Figure 1 As shown, a capsule endoscopy path prediction method based on multimodal data fusion is provided, comprising the following steps: step 110, step 120 and step 130. The method flow steps are only a possible implementation of the present invention.

[0050] Step 110: collect the patient's body position time series data in the current time period by using a plurality of inertial measurement units (IMUs), and collect the patient's digestive tract time series image data in the current time period by using a capsule endoscope.

[0051] Optionally, multiple inertial measurement units (IMUs) are placed on the patient's wrists and under the xiphoid process of the chest and abdomen; through the linkage of the IMUs under the wrists and xiphoid process, the posture changes of the patient's upper limbs and trunk can be fully captured to obtain more comprehensive body position information, avoiding positioning errors caused by position changes, respiratory interference, etc. of a single sensor. By fusing IMU data at multiple points, an accurate patient body position model can be generated, further supporting the accurate generation of path prediction and body position adjustment prompts.

[0052] Optionally, multiple IMUs are used to capture the posture time series data of multiple parts of the patient, such as the upper limbs and torso.

[0053] Optionally, multiple IMUs are used to collect acceleration data and angular velocity data of the patient at a high frequency (such as 50 Hz) in the current time period, and the acceleration data and angular velocity data are fused through Kalman filtering or complementary filtering algorithm to calculate the pitch angle (Pitch), roll angle (Roll), inclination angle (Inclination Angle) and yaw angle (Yaw).

[0054] Optionally, within the current time period, the patient's movements are marked, such as supine, left side lying, left half-supported, right half-supported, prone, right side lying, right half side, left half-prone, standing, etc.

[0055] It can be understood that collecting the patient's posture time series data by multiple inertial measurement units IMUs in the current time period and collecting the patient's digestive tract time series image data by capsule endoscopy in the current time period provide more comprehensive multimodal data for subsequent path prediction.

[0056] Step 120 : preprocess the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data.

[0057] Optionally, the body position time series data and time series image data are preprocessed by data cleaning, denoising, normalization, interpolation, etc.

[0058] In some embodiments, preprocessing the body position time series data and the time series image data includes:

[0059] Perform noise filtering and normalization preprocessing on body position time series data, and perform standardization and data enhancement preprocessing on time series image data;

[0060] The dynamic time warping (DTW) algorithm is used to align the body position time series data and the time series image data.

[0061] Among them, the Dynamic Time Warping (DTW) algorithm is an algorithm used to measure the similarity between two time series and to calculate the optimal matching distance between the two time series.

[0062] Optionally, a dynamic time warping (DTW) algorithm is used to calculate the cumulative distance between the body position time series data and the time series image data; based on the cumulative distance, the optimal alignment path of the body position time series data and the time series image data is determined; based on the optimal alignment path, the body position time series data and the time series image data are aligned.

[0063] It can be understood that by aligning the body position time series data and the time series image data, it can be ensured that the two can be synchronized on the time axis.

[0064] Step 130: input the preprocessed body position time series data and the preprocessed time series image data into a pre-trained prediction model to obtain the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the prediction model;

[0065] The prediction model is trained based on the preprocessed sample posture time series data and the preprocessed sample time series image data, as well as the path label of the sample capsule endoscope and the posture adjustment prompt label of the sample patient.

[0066] Among them, the path prediction results include the predicted probability of the part of the digestive tract that the capsule endoscope is about to cover. Based on the predicted probability, posture adjustment prompts can be generated in real time to ensure that the capsule endoscope can capture the uncovered area.

[0067] It should be noted that the prediction model can predict the possible motion trajectory of the capsule endoscope in multiple time steps in the future. For example, through continuous path prediction for 5 seconds (2 Hz per second), the prediction model can provide the path distribution in the next 10 frames.

[0068] Optionally, a path smoothing technique, such as Kalman filtering or sliding average, is introduced into the prediction model to ensure the consistency and accuracy of the path prediction.

[0069] Among them, the patient's position adjustment prompt is used to guide the patient to adjust the position so that the capsule endoscope can capture the uncovered area.

[0070] Optionally, the posture adjustment prompt includes adjusting the angle, such as "Please lie on your left side 20°".

[0071] Optionally, the relative position between the patient's current body position and the target area is calculated to obtain an adjustment angle.

[0072] Optionally, the calculation formula for adjusting the angle is as follows:

[0073] ;

[0074] in, To adjust the angle, is the angle of the target area, is the patient's current angle.

[0075] Optionally, the body position adjustment prompt includes a step-by-step adjustment prompt. For example, when it is detected that the patient needs to adjust from a standing position to a side-lying position, the body position adjustment prompt is "Please adjust to a standing position first, slightly rotate 15° to the right, and then slowly transition to a right side-lying position"; through staged prompts, the patient can be helped to adjust gradually without losing balance.

[0076] Optionally, the imaged and unimaged areas of the digestive tract are determined based on the image features, and a position adjustment prompt for the patient is generated in combination with the capsule endoscopy path prediction result and the patient's current position.

[0077] For example, if it is determined that the greater curvature of the gastric fundus is not imaged, the patient is prompted to adjust the body position to ensure that the area is fully covered.

[0078] For example, the path prediction result of the capsule endoscope is "move to the gastric body area", and the corresponding posture adjustment prompt is "please lie on your right side".

[0079] The path prediction results include the predicted probabilities of the digestive tract sites that the capsule endoscope will cover, such as P (gastric body) = 0.6, P (gastric fundus) = 0.3, and P (gastric antrum) = 0.1.

[0080] Optionally, based on the prediction model, according to the time series characteristics of the preprocessed posture time series data, the path prediction results of the capsule endoscopy and the patient's posture adjustment prompts are generated, and according to the image characteristics of the preprocessed sample time series image data, the path prediction results are optimized to obtain optimized path prediction results.

[0081] Figure 2 The second flow chart of the capsule endoscopy path prediction method based on multimodal data fusion provided by the embodiment of the present invention. Figure 2 As shown, a capsule endoscopy path prediction method based on multimodal data fusion is provided, comprising the following steps:

[0082] Acquire IMUs sensor data and capsule endoscopy images;

[0083] Input the IMUs sensor data into the time series processing module to obtain the time series features output by the time series processing module;

[0084] Inputting the capsule endoscopy image into the image feature extraction module to obtain the image features output by the image feature extraction module;

[0085] The time series features and image features are input into the multimodal fusion module to obtain the path prediction results and posture adjustment prompts output by the multimodal fusion module.

[0086] Among them, IMUs sensor data refers to the patient's body position data collected by multiple IMUs, such as acceleration, angular velocity, etc.; capsule endoscopy image refers to the image of the patient's digestive tract collected by capsule endoscopy.

[0087] Optionally, the multimodal fusion module is used to fuse the time series features and the image features to obtain fused features, and perform path prediction and posture adjustment prompt prediction based on the fused features.

[0088] The image features at least include the current position of the capsule endoscope.

[0089] Optionally, path prediction is performed according to the time series characteristics and the current position of the capsule endoscope to obtain a path prediction result.

[0090] Optionally, a posture adjustment prompt is obtained based on the path prediction result and the patient's current posture data.

[0091] The path prediction result at least includes the probability distribution of the target partition, and the posture adjustment prompt at least includes the adjustment angle and the action category.

[0092] In some embodiments, the prediction model includes a body position feature extraction layer, an image feature extraction layer, a feature fusion layer, and an output layer.

[0093] Figure 3 A flow chart of inputting preprocessed posture time series data and preprocessed time series image data into a pre-trained prediction model is provided in an embodiment of the present invention. Figure 3 As shown, the preprocessed posture time series data and the preprocessed time series image data are input into the pre-trained prediction model to obtain the capsule endoscopy path prediction result and the patient's posture adjustment prompt output by the prediction model, including:

[0094] Step 310: input the preprocessed body position time series data into the body position feature extraction layer to obtain the time series features output by the body position feature extraction layer;

[0095] Step 320: input the preprocessed time series image data into the image feature extraction layer to obtain the image features output by the image feature extraction layer;

[0096] Step 330: input the time series features and the image features into the feature fusion layer to obtain the fusion features output by the feature fusion layer;

[0097] Step 340: Input the fused features to the output layer, and obtain the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the output layer.

[0098] Among them, the time series features at least include three-axis acceleration, three-axis angular velocity and attitude angle.

[0099] It should be noted that each frame of image data corresponds to body position time series data within a time window.

[0100] Optionally, the feature fusion layer uses a dynamic weighting mechanism to fuse time series features and image features. The calculation formula of the fusion feature is as follows:

[0101] ;

[0102] ;

[0103] in, To fusion features, is the time series feature, is the weight of the time series feature, is the image feature, is the confidence of the preprocessed posture time series data, is the confidence of the preprocessed time series image data.

[0104] Optionally, the path distribution of multiple future time steps is predicted based on the fused features of multiple current time steps.

[0105] Optionally, the image feature extraction layer uses adaptive pooling technology to process image data of different sizes to ensure that the output feature map dimensions are consistent, and the extracted high-order image features are mapped to the probability distribution of partition classification through the fully connected layer.

[0106] Optionally, feature extraction is performed on the preprocessed time-series image data, and a two-level partitioning strategy of overall partitioning and local partitioning is adopted to identify path coverage and uncovered areas, such as accurately positioning the captured local partitions within the large stomach area, such as small target areas such as the cardia and pylorus.

[0107] It can be understood that by adopting a two-level partitioning strategy to perform fine partitioning of the image, the unphotographed area can be located more accurately, avoiding missed detection by capsule endoscopy.

[0108] In some embodiments, the body position feature extraction layer includes a Transformer layer based on a multi-head self-attention mechanism, the image feature extraction layer includes a lightweight convolutional neural network CNN layer, and the image features include refined regional classification results of preprocessed time series image data.

[0109] Optionally, the long-distance dependencies of the preprocessed body position time series data are captured based on the Transformer layer, and the time series features are output.

[0110] Optionally, the body position feature extraction layer includes a Transformer layer or other time series processing layer based on a multi-head self-attention mechanism to capture long-term dependencies in body position time series data.

[0111] Optionally, the image feature extraction layer includes a lightweight neural network CNN layer or other lightweight image processing models to adapt to real-time processing in a low computing resource environment.

[0112] It should be noted that, according to needs, the levels of the body position feature extraction layer and the image feature extraction layer can be expanded, for example, by adding additional convolutional layers, attention mechanisms, etc., to adapt to different usage scenarios and data complexity.

[0113] It can be understood that by fusing time series features and image features, the instability caused by a single data source can be effectively overcome and the stability and accuracy of path prediction can be improved.

[0114] In an embodiment of the present invention, multiple inertial measurement units IMUs are used to collect the patient's body position time series data within the current time period, and a capsule endoscope is used to collect the patient's digestive tract time series image data within the current time period; the body position time series data and the time series image data are preprocessed to obtain preprocessed body position time series data and preprocessed time series image data; the preprocessed body position time series data and the preprocessed time series image data are input into a pre-trained prediction model to obtain the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the prediction model, thereby improving the accuracy of capsule endoscope path prediction and effectively avoiding missed detection.

[0115] Figure 4 The present invention provides a flow chart of continuously monitoring the patient's body position changes and the image data collected by the capsule endoscope. Figure 4 As shown, in some embodiments, the above method also includes:

[0116] After the patient adjusts his / her body position, the patient's body position change and image data collected by the capsule endoscope are continuously monitored to obtain monitoring data, and new body position adjustment prompts are generated based on the monitoring data.

[0117] It should be noted that the patient's adjusted position data and the capsule endoscope's path changes are continuously monitored, and the adjustment effect is dynamically evaluated. If the target area is not fully covered, the path prediction and adjustment strategy are updated to generate new position prompts, forming a complete closed-loop feedback process.

[0118] In some embodiments, after the patient adjusts his / her body position, the patient's body position change and the image data collected by the capsule endoscope are continuously monitored to obtain monitoring data, and a new body position adjustment prompt is generated according to the monitoring data, including:

[0119] After the patient has adjusted his / her body position, the patient's body position monitoring data is obtained, and based on the body position monitoring data, it is determined whether the patient has completed the body position adjustment according to the body position adjustment prompt. If so, the body position adjustment is confirmed to be completed. Otherwise, a new body position adjustment prompt is generated to guide the patient to adjust his / her body position again, and the process is repeated until the body position adjustment is confirmed to be completed.

[0120] After confirming that the position adjustment is completed, obtain the image data of the digestive tract collected by the capsule endoscope, and determine whether the target area is photographed based on the image data. If so, stop the instruction; otherwise, generate a position adjustment suggestion so that the capsule endoscope can photograph the target area.

[0121] It is understandable that by adopting a closed-loop feedback mechanism, the patient's position can be continuously optimized, the optimal adjustment strategy can be dynamically generated, and each adjustment can be ensured to be confirmed and further optimized, which can avoid problems such as missed detection and path deviation, and significantly reduce the number of adjustments and inspection time.

[0122] In some embodiments, the above method further comprises:

[0123] Acquire the patient's physiological information and adaptively adjust the patient's posture adjustment prompts based on the patient's physiological information.

[0124] Among them, physiological information at least includes information such as height, weight, and physiological conditions.

[0125] It is understandable that by acquiring the patient's physiological information and adaptively adjusting the patient's body position adjustment prompt according to the patient's physiological information, personalized body position adjustment prompts can be obtained, thereby improving user experience.

[0126] Figure 5 A schematic diagram of a process of determining a prediction model provided by an embodiment of the present invention. In some embodiments, the process of determining a prediction model includes:

[0127] Step 510: collecting sample body position time series data of the sample patient within the sample time period through multiple IMUs, and collecting sample time series image data of the digestive tract of the sample patient within the sample time period through a sample capsule endoscope;

[0128] Step 520: preprocess the sample body position time series data and the sample time series image data to obtain preprocessed sample body position time series data and preprocessed sample time series image data;

[0129] Step 530, determining a path label of a sample capsule endoscope and a body position adjustment prompt label of a sample patient;

[0130] Step 540: Use the preprocessed sample posture time series data and the preprocessed sample time series image data as training data, and use the path label of the sample capsule endoscope and the posture adjustment prompt label of the sample patient as sample labels to train the initial prediction model. After the training is completed, a prediction model is obtained.

[0131] Optionally, the initial prediction model includes an initial body position feature extraction layer, an initial image feature extraction layer, an initial feature fusion layer and an initial output layer.

[0132] In some embodiments, training the initial prediction model includes:

[0133] Inputting the preprocessed sample body position time series data and the preprocessed sample time series image data into the initial prediction model, obtaining the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient output by the initial prediction model;

[0134] Calculate the loss function value based on the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient, as well as the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient;

[0135] Based on the loss function value, the parameters of the initial prediction model are iteratively optimized to obtain the prediction model.

[0136] Optionally, a mean square error loss function is used to calculate a first loss function value based on the path prediction result of the sample capsule endoscope and the path label of the sample capsule endoscope.

[0137] Optionally, a cross entropy loss function is used to calculate a second loss function value based on the posture adjustment prompts of the sample patients and the posture adjustment prompt labels of the sample patients.

[0138] Optionally, based on the first loss function value and the second loss function value, and the weight of the first loss function value and the second loss function value, a total loss function value is calculated, and the parameters of the initial prediction model are iteratively optimized according to the total loss function value.

[0139] Optionally, use an optimizer combined with a dynamic learning rate to adjust model parameters and introduce Early Stopping to avoid overfitting.

[0140] Optionally, the preprocessed sample body position time series data and the preprocessed sample time series image data are input into the initial prediction model to obtain the sample capsule endoscopy path prediction result and the sample patient's body position adjustment prompt output by the initial prediction model, including:

[0141] Input the preprocessed sample body position time series data into the initial body position feature extraction layer to obtain the sample time series features output by the initial body position feature extraction layer;

[0142] Input the preprocessed sample time series image data into the initial image feature extraction layer to obtain the sample image features output by the initial image feature extraction layer;

[0143] Input the sample time series features and sample image features into the initial feature fusion layer to obtain the sample fusion features output by the initial feature fusion layer;

[0144] The sample fusion features are input into the initial output layer to obtain the path prediction results of the sample capsule endoscopy and the posture adjustment prompts of the sample patients output by the initial output layer.

[0145] Optionally, the initial body position feature extraction layer includes a Transformer layer based on a multi-head self-attention mechanism, the initial image feature extraction layer includes a lightweight convolutional neural network CNN layer, and the sample image features include refined regional classification results of preprocessed sample time series image data.

[0146] The following describes a capsule endoscope path prediction device based on multimodal data fusion provided in an embodiment of the present invention. The capsule endoscope path prediction device based on multimodal data fusion described below and the capsule endoscope path prediction method based on multimodal data fusion described above can refer to each other.

[0147] Figure 6 A schematic diagram of the structure of a capsule endoscopy path prediction device based on multimodal data fusion provided in an embodiment of the present invention is shown in FIG. Figure 6 As shown, the capsule endoscopy path prediction device 600 based on multimodal data fusion includes:

[0148] The data acquisition unit 610 is used to acquire the patient's body position time series data in the current time period through multiple inertial measurement units IMU, and to acquire the patient's digestive tract time series image data in the current time period through a capsule endoscope;

[0149] A preprocessing unit 620, configured to preprocess the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data;

[0150] The prediction unit 630 is used to input the preprocessed body position time series data and the preprocessed time series image data into a pre-trained prediction model to obtain the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the prediction model;

[0151] The prediction model is trained based on the preprocessed sample posture time series data and the preprocessed sample time series image data, as well as the path label of the sample capsule endoscope and the posture adjustment prompt label of the sample patient.

[0152] Optionally, the prediction model includes a body position feature extraction layer, an image feature extraction layer, a feature fusion layer and an output layer;

[0153] Correspondingly, the preprocessed posture time series data and the preprocessed time series image data are input into the pre-trained prediction model to obtain the capsule endoscopy path prediction result and the patient's posture adjustment prompt output by the prediction model, including:

[0154] Input the preprocessed body position time series data into the body position feature extraction layer to obtain the time series features output by the body position feature extraction layer;

[0155] Input the preprocessed time series image data into the image feature extraction layer to obtain the image features output by the image feature extraction layer;

[0156] Input the time series features and image features into the feature fusion layer to obtain the fusion features output by the feature fusion layer;

[0157] The fused features are input into the output layer to obtain the capsule endoscopy path prediction results and the patient's position adjustment prompts output by the output layer.

[0158] Optionally, the body position feature extraction layer includes a Transformer layer based on a multi-head self-attention mechanism, the image feature extraction layer includes a lightweight convolutional neural network CNN layer, and the image features include refined regional classification results of preprocessed time series image data.

[0159] Optionally, the capsule endoscopy path prediction device 600 based on multimodal data fusion further includes:

[0160] The monitoring unit is used to continuously monitor the patient's body position changes and image data collected by the capsule endoscope after the patient adjusts his body position, obtain monitoring data, and generate new body position adjustment prompts based on the monitoring data.

[0161] Optionally, after the patient adjusts his / her body position, the patient's body position change and the image data collected by the capsule endoscope are continuously monitored to obtain monitoring data, and a new body position adjustment prompt is generated according to the monitoring data, including:

[0162] After the patient has adjusted his / her body position, the patient's body position monitoring data is obtained, and based on the body position monitoring data, it is determined whether the patient has completed the body position adjustment according to the body position adjustment prompt. If so, the body position adjustment is confirmed to be completed. Otherwise, a new body position adjustment prompt is generated to guide the patient to adjust his / her body position again, and the process is repeated until the body position adjustment is confirmed to be completed.

[0163] After confirming that the position adjustment is completed, obtain the image data of the digestive tract collected by the capsule endoscope, and determine whether the target area is photographed based on the image data. If so, stop the instruction; otherwise, generate a position adjustment suggestion so that the capsule endoscope can photograph the target area.

[0164] Optionally, the capsule endoscopy path prediction device 600 based on multimodal data fusion further includes:

[0165] The adjustment unit is used to obtain the patient's physiological information and adaptively adjust the patient's body position adjustment prompt according to the patient's physiological information.

[0166] Optionally, preprocessing the body position time series data and the time series image data includes:

[0167] Perform noise filtering and normalization preprocessing on body position time series data, and perform standardization and data enhancement preprocessing on time series image data;

[0168] The dynamic time warping (DTW) algorithm is used to align the body position time series data and the time series image data.

[0169] Optionally, the process of determining the prediction model includes:

[0170] Collecting sample body position time-series data of the sample patient within the sample time period through multiple IMUs, and collecting sample time-series image data of the digestive tract of the sample patient within the sample time period through a sample capsule endoscope;

[0171] Preprocessing the sample body position time series data and the sample time series image data to obtain preprocessed sample body position time series data and preprocessed sample time series image data;

[0172] Determine the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient;

[0173] The preprocessed sample posture time series data and the preprocessed sample time series image data are used as training data, and the path label of the sample capsule endoscope and the posture adjustment prompt label of the sample patient are used as sample labels to train the initial prediction model. After the training is completed, the prediction model is obtained.

[0174] Optionally, train an initial prediction model, including:

[0175] Inputting the preprocessed sample body position time series data and the preprocessed sample time series image data into the initial prediction model, obtaining the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient output by the initial prediction model;

[0176] Calculate the loss function value based on the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient, as well as the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient;

[0177] Based on the loss function value, the parameters of the initial prediction model are iteratively optimized to obtain the prediction model.

[0178] It should be noted here that the capsule endoscopy path prediction device based on multimodal data fusion provided in an embodiment of the present invention can implement all the method steps implemented in the above-mentioned capsule endoscopy path prediction method embodiment based on multimodal data fusion, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0179] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A capsule endoscopy path prediction method based on multimodal data fusion, characterized in that: include: Collecting time-series data of the patient's body position in a current time period by using a plurality of inertial measurement units (IMUs), and collecting time-series image data of the patient's digestive tract in a current time period by using a capsule endoscope; Preprocessing the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data; Inputting the preprocessed body position time series data and the preprocessed time series image data into a pre-trained prediction model, and obtaining the capsule endoscope path prediction result and the patient's body position adjustment prompt output by the prediction model; The prediction model is obtained by training based on the preprocessed sample body position time series data and the preprocessed sample time series image data, as well as the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient; The prediction model includes a body position feature extraction layer, an image feature extraction layer, a feature fusion layer and an output layer; the body position feature extraction layer includes a Transformer layer based on a multi-head self-attention mechanism, the image feature extraction layer includes a lightweight convolutional neural network CNN layer, and the image feature includes a refined regional classification result of the preprocessed time series image data; The method further comprises: After the patient adjusts his / her body position, the changes in the patient's body position and the image data collected by the capsule endoscope are continuously monitored to obtain monitoring data, and a new body position adjustment prompt is generated based on the monitoring data.

2. The capsule endoscopy path prediction method based on multimodal data fusion according to claim 1, characterized in that: The preprocessed posture time series data and the preprocessed time series image data are input into a pre-trained prediction model to obtain the capsule endoscope path prediction result and the patient's posture adjustment prompt output by the prediction model, including: Inputting the preprocessed body position time series data into the body position feature extraction layer to obtain the time series features output by the body position feature extraction layer; Inputting the preprocessed time-series image data into the image feature extraction layer to obtain image features output by the image feature extraction layer; Inputting the time series features and the image features into the feature fusion layer to obtain fusion features output by the feature fusion layer; The fusion feature is input into the output layer, and the path prediction result of the capsule endoscope and the body position adjustment prompt of the patient are obtained as output by the output layer.

3. The capsule endoscopy path prediction method based on multimodal data fusion according to claim 1, characterized in that: After the patient adjusts the body position, continuously monitoring the body position change of the patient and the image data collected by the capsule endoscope to obtain monitoring data, and generating a new body position adjustment prompt according to the monitoring data, including: After the patient has adjusted the body position, the body position monitoring data of the patient is obtained, and according to the body position monitoring data, it is determined whether the patient has completed the body position adjustment according to the body position adjustment prompt, and if so, it is confirmed that the body position adjustment is completed, otherwise, a new body position adjustment prompt is generated to guide the patient to adjust the body position again, and the cycle is repeated until it is confirmed that the body position adjustment is completed; After confirming that the body position adjustment is completed, the image data of the digestive tract collected by the capsule endoscope is obtained, and it is determined whether the target area is photographed based on the image data. If so, the instruction is stopped; otherwise, a body position adjustment suggestion is generated so that the capsule endoscope can photograph the target area.

4. The capsule endoscopy path prediction method based on multimodal data fusion according to claim 1, characterized in that: The method further comprises: Acquire the physiological information of the patient, and adaptively adjust the patient's body position adjustment prompt according to the patient's physiological information.

5. The capsule endoscopy path prediction method based on multimodal data fusion according to any one of claims 2 to 4, characterized in that: The preprocessing of the body position time series data and the time series image data includes: Performing noise filtering and normalization preprocessing on the body position time series data, and performing standardization and data enhancement preprocessing on the time series image data; The body position time series data and the time series image data are aligned by using a dynamic time warping (DTW) algorithm.

6. The capsule endoscopy path prediction method based on multimodal data fusion according to claim 1, characterized in that: The process of determining the prediction model includes: Collecting sample body position time-series data of a sample patient within a sample time period through a plurality of IMUs, and collecting sample time-series image data of the digestive tract of the sample patient within a sample time period through a sample capsule endoscope; Preprocessing the sample body position time series data and the sample time series image data to obtain preprocessed sample body position time series data and preprocessed sample time series image data; Determining a path label of the sample capsule endoscope and a body position adjustment prompt label of the sample patient; The preprocessed sample posture time series data and the preprocessed sample time series image data are used as training data, and the path label of the sample capsule endoscope and the posture adjustment prompt label of the sample patient are used as sample labels to train an initial prediction model. After the training is completed, the prediction model is obtained.

7. The capsule endoscopy path prediction method based on multimodal data fusion according to claim 6, characterized in that: The training of the initial prediction model comprises: Inputting the preprocessed sample body position time series data and the preprocessed sample time series image data into an initial prediction model, and obtaining the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient output by the initial prediction model; Calculating a loss function value based on the path prediction result of the sample capsule endoscope and the body position adjustment prompt of the sample patient, as well as the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient; Based on the loss function value, the parameters of the initial prediction model are iteratively optimized to obtain the prediction model.

8. A capsule endoscopy path prediction device based on multimodal data fusion, characterized in that: include: A data acquisition unit, used to acquire the patient's body position time series data in the current time period through a plurality of inertial measurement units (IMUs), and to acquire the patient's digestive tract time series image data in the current time period through a capsule endoscope; A preprocessing unit, used for preprocessing the body position time series data and the time series image data to obtain preprocessed body position time series data and preprocessed time series image data; A prediction unit, used for inputting the preprocessed posture time series data and the preprocessed time series image data into a pre-trained prediction model, and obtaining the capsule endoscope path prediction result and the patient's posture adjustment prompt output by the prediction model; The prediction model is obtained by training based on the preprocessed sample body position time series data and the preprocessed sample time series image data, as well as the path label of the sample capsule endoscope and the body position adjustment prompt label of the sample patient; The prediction model includes a body position feature extraction layer, an image feature extraction layer, a feature fusion layer and an output layer; the body position feature extraction layer includes a Transformer layer based on a multi-head self-attention mechanism, the image feature extraction layer includes a lightweight convolutional neural network CNN layer, and the image feature includes a refined regional classification result of the preprocessed time series image data; The device also includes: The monitoring unit is used to continuously monitor the patient's body position changes and the image data collected by the capsule endoscope after the patient adjusts his body position, obtain monitoring data, and generate new body position adjustment prompts based on the monitoring data.

Images