A gastric environment detection method and system based on a capsule robot

Abstract

The application discloses a kind of gastric environment detection method and system based on capsule robot, by obtaining the pose of capsule robot in stomach, and according to pose and magnetic sensor data constructs the positioning model of capsule robot, obtains the PH data collected by PH sensor, and according to PH data and positioning model controls capsule endoscope to collect multiple stomach images, carries out image pre-processing to multiple stomach images to obtain capsule endoscope image, and extracts the image feature information of capsule endoscope image, image feature information is input into the trained deep neural network and is trained to obtain image classification result, based on image classification result and PH data determine gastric environment detection result, using the capsule robot with capsule gastroscope and multiple sensors, gastric environment can be visualized and accurately positioned, improve the accuracy of PH value detection and clearly observe the stomach, provide prevention and timely treatment for stomach-related diseases.

Description

Technical Field

[0001] This invention belongs to the field of capsule detection technology, and in particular relates to a method and system for detecting the gastric environment based on a capsule robot. Background Technology

[0002] In modern dairy farming, subacute rumen acidosis is a common gastrointestinal disease. To increase milk production and thus achieve higher economic returns, concentrated feed is widely used in large-scale dairy farming. During the degradation of concentrated feed in the rumen, it produces large amounts of lactic acid and other substances, causing a decrease in the pH value of the rumen fluid. If the rumen pH value remains below 5.6 for more than three hours per day or below 5.8 for more than five to six hours per day, subacute rumen acidosis will occur. Therefore, long-term rumen pH monitoring equipment for dairy cows remains a hot topic in intelligent livestock farming.

[0003] Currently, most methods involve implanting capsules into the cow's stomach to detect data such as body temperature, posture, and activity level. This data can be used to predict the cow's health status. However, these methods lack precise localization of the cow's stomach environment and are not easily visualized for analysis, thus affecting the accuracy of stomach environment detection and disease prediction. Therefore, there is an urgent need to provide a stomach environment detection method based on capsule robots to solve the aforementioned technical problems. Summary of the Invention

[0004] In view of this, the present invention provides a method and system for gastric environment detection based on a capsule robot. The capsule robot, equipped with a capsule endoscope and multiple sensors, can perform visual detection and precise positioning of the gastric environment, improving the accuracy of pH value detection and clear observation of the gastric condition. The specific technical solution is as follows.

[0005] In a first aspect, the present invention provides a method for detecting the gastric environment based on a capsule robot, wherein the capsule robot includes a capsule endoscope, a magnetic shell disposed on the capsule endoscope, and at least two sensors connected to the capsule endoscope, wherein the at least two sensors include a magnetic sensor and a pH sensor, and the gastric environment detection method includes: The pose of the capsule robot in the stomach is obtained, and a localization model of the capsule robot is constructed based on the pose and magnetic sensor data, wherein the pose includes the position information and attitude information of the magnetic shell; Acquire pH data collected by a pH sensor, and control the capsule endoscope to acquire multiple gastric images based on the pH data and the positioning model; Image preprocessing is performed on the multiple gastric images to obtain capsule endoscopy images, and image feature information of the capsule endoscopy images is extracted; The image feature information is input into a trained deep neural network to obtain image classification results, and the gastric environment detection results are determined based on the image classification results and pH data.

[0006] As a preferred embodiment of the above technical solution, the pose of the capsule robot in the stomach is obtained, and a positioning model of the capsule robot is constructed based on the pose and magnetic sensor data, including: Select a point P in the space containing the magnetic shell, where the distance between P and the magnetic shell is greater than the diameter of the magnetic shell. Calculate the magnetic flux density at point P using the formula for the magnetic flux density of a magnetic dipole. Here, the length of the magnetic shell is L, the radius is r, and the surface of the magnetic shell has a uniform magnetization. The magnetic moment of a magnetic dipole is expressed as ,use To represent the standard direction vector of the magnetic field of the magnetic shell, Representing a spatial point Magnetic flux density at: (1) (2) (3) (4) in, This represents the relative permeability of a medium, where the medium refers to air. , Indicates the magnetic permeability in a vacuum; Indicates the position point from the magnetic shell to space point The vector, The magnetic field strength of a magnetic dipole is represented by the formula for the distance from the magnetic shell's position point to a point in space. , This represents the magnetic moment corresponding to the small circulation; When there are N sensors, the first... The position of each magnetic sensor in space is represented as follows: , No. The magnetic flux density measured by each magnetic sensor is: (5) According to formula (1), the magnetic induction intensity Decomposed into components in three orthogonal directions: (6) (7) (8) According to the direction vector of the magnetic shell get The cross product of both sides of formula (1) Dot product on both sides get And simplify to the linear form of the equation as follows ,in:

[0007]

[0008]

[0009] in, , ,exist In the equation, and Based on known magnetic sensor data and the position parameters of the magnetic sensor The magnetic induction intensity values ​​measured by five magnetic sensors were calculated to obtain a matrix. and a vector ,get Solve for R and then solve the simultaneous equations. Substituting the values ​​into the elimination function yields the positioning parameters. The value of the positioning parameter. The value is used as the pose of the magnetic shell.

[0010] As a preferred embodiment of the above technical solution, geomagnetic field data is obtained, and magnetic field data are collected twice before and after the magnetic shell rotates. Based on the position coordinates and measurement data of a preset number of magnetic sensors, the position information of the magnetic shell is solved using the linear least squares method. ; Based on positioning parameters and location information Obtain attitude information The positioning model of the capsule robot is determined based on positioning parameters, attitude information, and position information.

[0011] As a preferred embodiment of the above technical solution, acquiring pH data from a pH sensor and controlling the capsule endoscope to acquire multiple gastric images based on the pH data and the positioning model includes: The pH sensor probe uses a piston-driven mechanism to pressurize the internal reference electrolyte. The power required for piston propulsion is provided by a spring. After deformation, the inner hole of the silicone tube of the pH sensor probe becomes circular. Therefore, the outflow of electrolyte inside the silicone tube can be calculated using the Hagen-Poiseuille law. (9) Where Q represents traffic. It is the inner diameter of the silicone tube after deformation. It is the pressure difference per unit length. It is viscosity; the inflow pressure, outflow pressure, and flow rate of the silicone tube are obtained by analyzing the external flow rate of the internal reference electrolyte to obtain the value in formula (9). ,but The expression is: (10) in, It refers to the stiffness of the compression spring. It is the piston outer diameter. It is the initial volume of the internal reference solution. This refers to the volume of internal reference solution that flows out after the valve on the silicone tube has been open for a first preset time. The piston is powered by the spring.

[0012] As a preferred embodiment of the above technical solution, with the increase in probe usage time, the spring contraction decreases, and the pressure between the internal reference electrolyte and the external environment decreases. The integral form is: (11) in, It is the volume of the internal reference solution that flows out when the pressure of the internal reference solution drops to 5 kPa. It is the internal reference ratio to the initial pressure of the electrolyte. After the second preset duration of continuous operation, the internal pressure of the reference solution is: .

[0013] As a preferred embodiment of the above technical solution, image preprocessing is performed on the multiple gastric images to obtain capsule endoscopy images, and image feature information of the capsule endoscopy images is extracted, including: Obtain a window centered on the edge pixels of the highlight region in the capsule endoscopy image, and calculate the sum of the ratios of the R and B channels of the known pixels in the RGB color space, where the sum of the ratios represents the proportion of the known pixels. The product of the gradient normal vector and the iso-illuminance line in the specular resolution of the Criminisi algorithm is used as the data item, where the data line indicates that there is a clear intersection boundary between the known area and the specular area; The confidence level is added to the weight of the data item to calculate the priority at the highlight edge, and the edge pixel with the highest priority is obtained. Variance adjustment of sample block size is used to repair highlight areas, and the best matching block is obtained by combining RGB color channel examples and pixel examples to repair the highest priority pixel window. The pixels in the best matching block are copied to the highlight region in the highest priority sample block, and the highlight region mask matrix is ​​updated until the mask matrix is ​​0. The boundary pixels of the highlight region are replaced with the average value of all pixels in the window centered on the boundary pixels to complete the highlight denoising of the capsule endoscopy image.

[0014] As a preferred embodiment of the above technical solution, the product of the gradient normal vector of specular resolution and the isoluminance line in the Criminisi algorithm is used as the data item, including: Preset For capsule endoscopy images that need repair, The damaged area in the capsule endoscopy image. The damaged area in the capsule endoscopy image. The boundary between the known area and the damaged area. The pixel with the highest current priority value. For The central window, for The direction of the isoil lines at the point The Criminisi algorithm uses a gradient normal vector orthogonal to the boundary at each pixel. The priority calculation expression is: (12) in, for The confidence level of a point represents the window The expression for the proportion of pixels in a known region is: ; The larger the value, the more known area information it contains, and the more priority it will be for repair. for The data item of a point represents gradient normal vector of a point With isolux lines The expression for the product is: The larger the value, the more likely it is to indicate a clear boundary between the known and damaged areas, thus prioritizing repair. For window area, Normalization factor; Use window It is represented by the ratio of the R channel to the B channel of a known region of pixels. The higher the confidence level of a point, the greater the ratio of the R channel to the B channel. The more valid information a point has, the better the confidence level will be. The expression is: (13) in, This represents the ratio of the R channel to the B channel of pixel q. , and This represents the R and B channel values ​​in the RGB color space at point q; When the isolux line With gradient normal vector When vertical, data items If the value is 0, rewrite formula (12) as follows: ,in, These are the weighting coefficients.

[0015] As a preferred embodiment of the above technical solution, the capsule endoscopy image is preprocessed in the HSV color space, and the image is decomposed from the color space into hue, saturation, and brightness channels to separate color and brightness information, including: The extracted luminance channel image is subjected to adaptive CLAHE processing, and the luminance channel image is divided into multiple sub-regions, where each sub-region is of equal size, continuous, and non-overlapping. The histogram and required cropping value for each sub-region are calculated, and the corresponding expressions are as follows: (14) Where N represents the number of pixels in the sub-region, and M represents the number of gray levels contained in the corresponding sub-region. The cropping factor is calculated adaptively based on the weighted average of the image's grayscale mean and standard deviation. The relationship between gray values ​​and their corresponding cumulative distribution function (CDF) is fitted using a linear interpolation algorithm as follows: ,in, The calculation expression is , Let n represent the number of pixels with a grayscale value of n, and T represent the total number of pixels in the image. The calculation expression is: (15) in, For coefficient factors, The average gray level of the image pixels. The standard deviation of the image pixels is used to plot the gray-level histogram of the sub-region using the calculated cropping value. Perform cropping and calculate the total number of pixels exceeding the cropping value. The corresponding expression is: (16) The portion of pixels exceeding the clipping value is evenly distributed across all gray levels, with the number of pixels evenly distributed across each gray level set to [value missing]. The corresponding expression is Histogram after redistribution The calculation expression is: (17) The new histogram of each sub-region is equalized sequentially, and the final image is obtained through linear interpolation.

[0016] As a preferred embodiment of the above technical solution, the image feature information is input into a trained deep neural network for training to obtain image classification results. Based on the image classification results and pH data, the gastric environment detection results are determined, including: The deep neural network employs a Transform structure. The multi-head attention mechanism maps input data to multiple sub-regions, enabling the model to learn various feature representations of the input data from different sub-regions. In each attention head, the model calculates attention weights based on the similarity between query Q, key K, and value V, and uses these weights to perform a weighted average of the similarity values ​​to capture local and global feature information. The expression for the self-attention mechanism is: (18) in, , , These represent the query, key, and value, respectively. The key dimension is scaled by the dot product of the query and the key, and the attention weights obtained by normalization through the Softmax function are applied to the value V to obtain the weighted output result. If there are h attention heads, the output of the multi-attention mechanism is: (19) in, It is the output of the i-th head. It is the linear transformation matrix of the output layer; A feedforward neural network (FNN) is used to further perform a nonlinear transformation on the output of the attention layer. The FNN consists of two fully connected layers and an activation function. The preset input is U, and the output of the FNN is expressed as: (20) in, and It is a weight matrix. and It is the bias term, and ReLU is a non-linear activation function. ; A weighted average frame-level prediction aggregation function is used to calculate the final classification prediction by averaging the classification results of each frame. For each class j, the prediction score for each frame is calculated first. And calculate the corresponding weights based on the score of each frame. Weight This represents the contribution of the current frame to the final classification result. The expression for the weighted average aggregation function is: (twenty one) in, For the final weighted classification result, This represents the weight of frame t in category j. Let represent the prediction score for frame t, and the weight expression for each frame is: , .

[0017] Secondly, the present invention also provides a gastric environment detection system based on a capsule robot, applied to the aforementioned gastric environment detection method based on a capsule robot, comprising: The model building module is used to obtain the pose of the capsule robot in the stomach and build a localization model of the capsule robot based on the pose and magnetic sensor data, wherein the pose includes the position information and attitude information of the magnetic shell; The image acquisition module is used to acquire pH data collected by the pH sensor, and control the capsule endoscope to acquire multiple stomach images based on the pH data and the positioning model; The feature extraction module is used to preprocess the multiple gastric images to obtain capsule endoscopy images and extract the image feature information of the capsule endoscopy images; The classification and detection module is used to input the image feature information into a trained deep neural network to obtain image classification results, and to determine the gastric environment detection results based on the image classification results and pH data.

[0018] This invention provides a method and system for gastric environment detection based on a capsule robot. The method involves acquiring the capsule robot's pose in the stomach, constructing a positioning model based on the pose and magnetic sensor data, acquiring pH data from a pH sensor, and controlling the capsule endoscope to acquire multiple stomach images based on the pH data and the positioning model. Image preprocessing is performed on these stomach images to obtain capsule endoscope images, and image feature information is extracted from these images. This image feature information is then input into a trained deep neural network to obtain image classification results. Based on the image classification results and pH data, the gastric environment detection result is determined. Using a capsule robot equipped with a capsule endoscope and multiple sensors, the gastric environment can be visualized and precisely located, improving the accuracy of pH value detection and providing clear observation of the stomach condition, thus enabling prevention and timely treatment of gastric-related diseases. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of the gastric environment detection method based on a capsule robot provided by the present invention; Figure 2 A flowchart for image specular denoising provided by the present invention; Figure 3 The structural block diagram of the gastric environment detection system based on a capsule robot provided by the present invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0022] See Figure 1 This invention provides a method for detecting the gastric environment based on a capsule robot. The capsule robot includes a capsule endoscope, a magnetic shell disposed on the capsule endoscope, and at least two sensors connected to the capsule endoscope. The at least two sensors include a magnetic sensor and a pH sensor. The gastric environment detection method includes: S1: Obtain the pose of the capsule robot in the stomach, and construct a localization model of the capsule robot based on the pose and magnetic sensor data, wherein the pose includes the position information and attitude information of the magnetic shell; S2: Acquire pH data collected by the pH sensor, and control the capsule endoscope to acquire multiple gastric images based on the pH data and the positioning model; S3: Perform image preprocessing on the multiple stomach images to obtain capsule endoscopy images, and extract image feature information from the capsule endoscopy images; S4: Input the image feature information into the trained deep neural network to obtain the image classification result, and determine the gastric environment detection result based on the image classification result and PH data.

[0023] In this embodiment, the capsule endoscope can be a miniature endoscope or a CCD, mounted on the magnetic shell of the capsule robot. The capsule robot connects to the computer via wireless communication methods such as Bluetooth or LoRa to achieve real-time data acquisition and uploading. The capsule robot's pose in the stomach is acquired, and a positioning model of the capsule robot is constructed based on the pose and magnetic sensor data. This includes selecting a point P in the space where the magnetic shell is located, where the distance between point P and the magnetic shell is greater than the diameter of the magnetic shell; calculating the magnetic induction intensity at point P using the magnetic dipole magnetic induction intensity formula; where the length of the magnetic shell is L, the radius is r, and the surface of the magnetic shell has uniform magnetization. The magnetic moment of a magnetic dipole is expressed as ,use To represent the standard direction vector of the magnetic field of the magnetic shell, Representing a spatial point Magnetic flux density at: (1) (2) (3) (4) in, This represents the relative permeability of a medium, where the medium refers to air. , Indicates the magnetic permeability in a vacuum; Indicates the position point from the magnetic shell to space point The vector, The magnetic field strength of a magnetic dipole is represented by the formula for the distance from the magnetic shell's position point to a point in space. , This represents the magnetic moment corresponding to the small circulation; When there are N sensors, the first... The position of each magnetic sensor in space is represented as follows: , No. The magnetic flux density measured by each magnetic sensor is: (5) According to formula (1), the magnetic induction intensity Decomposed into components in three orthogonal directions: (6) (7) (8) According to the direction vector of the magnetic shell get The cross product of both sides of formula (1) Dot product on both sides get And simplify to the linear form of the equation as follows ,in:

[0024]

[0025]

[0026] in, , ,exist In the equation, and Based on known magnetic sensor data and the position parameters of the magnetic sensor The magnetic induction intensity values ​​measured by five magnetic sensors were calculated to obtain a matrix. and a vector ,get Solve for R and then solve the simultaneous equations. Substituting the values ​​into the elimination function yields the positioning parameters. The value of the positioning parameter, and the positioning parameter The value is used as the pose of the magnetic shell.

[0027] It should be noted that while linear algorithms can quickly calculate unknown parameters without the need for auxiliary initial values, they are easily limited by the current state of the data and the assumptions made. They are also sensitive to noise and outliers in the data, and the formulas... The solution can be obtained using data from only five triaxial magnetometers. Using more triaxial magnetometers and combining them with a nonlinear optimization algorithm can achieve even higher positioning accuracy. The preset total number of magnetometers used is... Given a triaxial magnetic sensor, the objective of the solution is to minimize... ,So , where the matrix The element is represented as ,matrix The elements are The optimal solution for R can be found using the constrained linear least squares method. According to the formula... get ,according to and The obtained R can be used to derive the attitude information of the magnetic shell. , The position information of the magnetic shell can be further derived. The expression is The expressions for parameters a and b with respect to c are obtained as follows: .

[0028] It should be understood that the magnetic field distribution of a magnetic dipole is similar to that of an electric dipole in the far field. The positioning optimization algorithm used in this invention is mainly for optimizing the position information of the capsule endoscope. By acquiring the pose of the capsule robot in the stomach, and constructing a positioning model of the capsule robot based on the pose and magnetic sensor data, the pH data collected by the pH sensor is acquired. Based on the pH data and the positioning model, the capsule endoscope is controlled to acquire multiple stomach images. Image preprocessing is performed on these multiple stomach images to obtain capsule endoscope images, and image feature information is extracted from the capsule endoscope images. This image feature information is input into a trained deep neural network for training to obtain image classification results. Based on the image classification results and pH data, the stomach environment detection results are determined. Using a capsule robot equipped with a capsule endoscope and multiple sensors, the stomach environment can be visualized and precisely located, improving the accuracy of pH value detection and providing clear observation of the stomach condition, thus offering prevention and timely treatment for stomach-related diseases.

[0029] Optionally, geomagnetic field data can be acquired, and magnetic field data can be collected twice before and after the magnetic shell rotates; Based on the position coordinates and measurement data of a preset number of magnetic sensors, the position information of the magnetic shell is solved using the linear least squares method. ; Based on positioning parameters and location information Obtain attitude information The positioning model of the capsule robot is determined based on positioning parameters, attitude information, and position information.

[0030] In this embodiment, pH data collected by a pH sensor is acquired, and the capsule endoscope is controlled to acquire multiple gastric images based on the pH data and the positioning model, including: The pH sensor probe uses a piston-driven mechanism to pressurize the internal reference electrolyte. The power required for piston propulsion is provided by a spring. After deformation, the inner hole of the silicone tube of the pH sensor probe becomes circular. Therefore, the outflow of electrolyte inside the silicone tube can be calculated using the Hagen-Poiseuille law. (9) Where Q represents traffic. It is the inner diameter of the silicone tube after deformation. It is the pressure difference per unit length. It is viscosity; the inflow pressure, outflow pressure, and flow rate of the silicone tube are obtained by analyzing the external flow rate of the internal reference electrolyte to obtain the value in formula (9). ,but The expression is: (10) in, It refers to the stiffness of the compression spring. It is the piston outer diameter. It is the initial volume of the internal reference solution. This refers to the volume of internal reference solution that flows out after the valve on the silicone tube has been open for a first preset time. The piston is powered by the spring.

[0031] It should be noted that as the probe's usage time increases, the spring contraction decreases, and the pressure difference between the internal reference electrolyte and the external environment decreases. The integral form is: (11) in, It is the volume of the internal reference solution that flows out when the pressure of the internal reference solution drops to 5 kPa. It is the internal reference ratio to the initial pressure of the electrolyte. After the second preset duration of continuous operation, the internal pressure of the reference solution is: .

[0032] Specifically, the pH sensor (pH measuring electrode) includes inner and outer shells, a valve component, a pressurizing component, and a measuring component. The outer shell consists of a magnetic shell and a rear shell, which are connected by threads. The rear shell has a rounded corner design, and the sharp edges of the wall stimulate the rumen tissue. The magnetic shell has 90-degree interval openings on its side to prevent the electrode from being unable to measure properly due to the presence of gas cavities in the rumen fluid. The inner shell includes a main body, a connecting tube, and a pressurizing rear cover. The two ends of the connecting tube are connected to the main body and the pressurizing rear cover by threads. The inner side of the inner shell contains a saturated KCl solution and a silver rod plated with silver chloride is inserted. Pt1000 and a glass electrode are fixed in the groove on the outer side. The two ends of the silicone tube are in contact with the internal reference electrolyte and the rumen fluid, respectively. The nickel alloy rod blocks the outflow of the internal reference electrolyte by squeezing the silicone tube. When the valve is opened, the BMF is energized and contracts, pulling the nickel alloy rod and releasing the silicone tube, allowing the internal reference electrolyte to flow out. The pressurizing components include a piston, spring, and inner tube. The measuring components, including the glass electrode, Ag / AgCl reference system, and PT1000, are all fixed on the main body of the inner shell, thereby improving the pH detection accuracy of the pH sensor.

[0033] Optionally, see Figure 2 The multiple gastric images are preprocessed to obtain capsule endoscopy images, and image feature information of the capsule endoscopy images is extracted, including: S10: Obtain a window centered on the edge pixels of the highlight region in the capsule endoscope image, and calculate the sum of the ratios of the R and B channels of the known pixels in the RGB color space, where the sum of the ratios represents the proportion of the known pixels. S11: The product of the gradient normal vector of the specular resolution in the Criminisi algorithm and the iso-illuminance line is used as the data item. The data line indicates that there is a clear intersection boundary between the known area and the specular area. S12: Add the confidence level to the weight of the data item to calculate the priority at the highlight edge and obtain the edge pixel with the highest priority; S13: Use variance-adjusted sample block size to repair highlight areas, and combine RGB color channel examples and pixel examples to obtain the best matching block to repair the highest priority pixel window; S14: Copy the pixels in the best matching block to the highlight region in the highest priority sample block, and update the highlight region Mask matrix until the Mask matrix is ​​0. Replace the boundary pixels of the highlight region with the average value of all pixels in the window centered on the boundary pixels to complete the highlight denoising of the capsule endoscopy image.

[0034] In this embodiment, the product of the gradient normal vector of specular resolution and the iso-illuminance line in the Criminisi algorithm is used as the data item, including: preset For capsule endoscopy images that need repair, The damaged area in the capsule endoscopy image. The damaged area in the capsule endoscopy image. The boundary between the known area and the damaged area. The pixel with the highest current priority value. For The central window, for The direction of the isoil lines at the point The Criminisi algorithm uses a gradient normal vector orthogonal to the boundary at each pixel. The priority calculation expression is: (12) in, for The confidence level of a point represents the window The expression for the proportion of pixels in a known region is: ; The larger the value, the more known area information it contains, and the more priority it will be for repair. for The data item of a point represents gradient normal vector of a point With isolux lines The expression for the product is: The larger the value, the more likely it is to indicate a clear boundary between the known and damaged areas, thus prioritizing repair. For window area, Normalization factor; Use window It is represented by the ratio of the R channel to the B channel of a known region of pixels. The higher the confidence level of a point, the greater the ratio of the R channel to the B channel. The more valid information a point has, the better the confidence level will be. The expression is: (13) in, This represents the ratio of the R channel to the B channel of pixel q. , and This represents the R and B channel values ​​in the RGB color space at point q; When the isolux line With gradient normal vector When vertical, data items If the value is 0, rewrite formula (12) as follows: ,in, These are weighting coefficients, which improve the accuracy of image processing.

[0035] It should be noted that the window size of the sample block in the Criminisi algorithm is a fixed 9 pixels. A 9-pixel window is recommended. When an image has rich detail, a smaller window should be used for more precise restoration; when the image has less detail, a larger window should be used to avoid block artifacts. Local variance effectively reflects the level of detail in an image; the larger the local variance, the richer the local detail.

[0036] Optionally, the capsule endoscopy image is preprocessed in the HSV color space, and the image is decomposed from the color space into hue, saturation, and lightness channels to separate color and brightness information, including: The extracted luminance channel image is subjected to adaptive CLAHE processing, and the luminance channel image is divided into multiple sub-regions, where each sub-region is of equal size, continuous, and non-overlapping. The histogram and required cropping value for each sub-region are calculated, and the corresponding expressions are as follows: (14) Where N represents the number of pixels in the sub-region, and M represents the number of gray levels contained in the corresponding sub-region. The cropping factor is calculated adaptively based on the weighted average of the image's grayscale mean and standard deviation. The relationship between gray values ​​and their corresponding cumulative distribution function (CDF) is fitted using a linear interpolation algorithm as follows: ,in, The calculation expression is , Let n represent the number of pixels with a grayscale value of n, and T represent the total number of pixels in the image. The calculation expression is: (15) in, For coefficient factors, The average gray level of the image pixels. The standard deviation of the image pixels is used to plot the gray-level histogram of the sub-region using the calculated cropping value. Perform cropping and calculate the total number of pixels exceeding the cropping value. The corresponding expression is: (16) The portion of pixels exceeding the clipping value is evenly distributed across all gray levels, with the number of pixels evenly distributed across each gray level set to [value missing]. The corresponding expression is Histogram after redistribution The calculation expression is: (17) The new histogram of each sub-region is equalized sequentially, and the final image is obtained through linear interpolation.

[0037] In this embodiment, the image feature information is input into a trained deep neural network for training to obtain image classification results. Based on the image classification results and pH data, the gastric environment detection results are determined. This includes: the deep neural network adopts a Transform structure, and the multi-head attention mechanism maps the input data to multiple sub-regions, enabling the model to learn multiple feature representations of the input data from different sub-regions. In each attention head, the model calculates attention weights based on the similarity between query Q, key K, and value V, and uses these weights to perform a weighted average of the similarity values ​​to capture local and global feature information. The expression for the self-attention mechanism is: (18) in, , , These represent the query, key, and value, respectively. The key dimension is scaled by the dot product of the query and the key, and the attention weights obtained by normalization through the Softmax function are applied to the value V to obtain the weighted output result. If there are h attention heads, the output of the multi-attention mechanism is: (19) in, It is the output of the i-th head. It is the linear transformation matrix of the output layer; A feedforward neural network (FNN) is used to further perform a nonlinear transformation on the output of the attention layer. The FNN consists of two fully connected layers and an activation function. The preset input is U, and the output of the FNN is expressed as: (20) in, and It is a weight matrix. and It is the bias term, and ReLU is a non-linear activation function. ; A weighted average frame-level prediction aggregation function is used to calculate the final classification prediction by averaging the classification results of each frame. For each class j, the prediction score for each frame is calculated first. And calculate the corresponding weights based on the score of each frame. Weight This represents the contribution of the current frame to the final classification result. The expression for the weighted average aggregation function is: (twenty one) in, For the final weighted classification result, This represents the weight of frame t in category j. Let represent the prediction score for frame t, and the weight expression for each frame is: , .

[0038] It should be noted that in the process of capsule endoscopy image classification, in addition to the extraction of local features, the model also needs to have the ability to model global information in order to capture long-range dependencies and improve the ability to identify complex lesion areas. The main goal of image preprocessing is to standardize capsule endoscopy images to ensure that images from different datasets have a consistent format and to improve the training stability of the model.

[0039] See Figure 3 The present invention also provides a gastric environment detection system based on a capsule robot, applied to the above-mentioned gastric environment detection method based on a capsule robot, comprising: The model building module is used to obtain the pose of the capsule robot in the stomach and build a localization model of the capsule robot based on the pose and magnetic sensor data, wherein the pose includes the position information and attitude information of the magnetic shell; The image acquisition module is used to acquire pH data collected by the pH sensor, and control the capsule endoscope to acquire multiple stomach images based on the pH data and the positioning model; The feature extraction module is used to preprocess the multiple gastric images to obtain capsule endoscopy images and extract the image feature information of the capsule endoscopy images; The classification and detection module is used to input the image feature information into a trained deep neural network to obtain image classification results, and to determine the gastric environment detection results based on the image classification results and pH data.

[0040] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0041] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0042] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for detecting the gastric environment based on a capsule robot, characterized in that, The capsule robot includes a capsule endoscope, a magnetic shell disposed on the capsule endoscope, and at least two sensors connected to the capsule endoscope, wherein the at least two sensors include a magnetic sensor and a pH sensor, and the gastric environment detection method includes: The pose of the capsule robot in the stomach is obtained, and a localization model of the capsule robot is constructed based on the pose and magnetic sensor data, wherein the pose includes the position information and attitude information of the magnetic shell; Acquire pH data collected by a pH sensor, and control the capsule endoscope to acquire multiple gastric images based on the pH data and the positioning model; Image preprocessing is performed on the multiple gastric images to obtain capsule endoscopy images, and image feature information of the capsule endoscopy images is extracted; The image feature information is input into a trained deep neural network to obtain image classification results, and the gastric environment detection results are determined based on the image classification results and pH data.

2. The method for detecting the gastric environment based on a capsule robot according to claim 1, characterized in that, Acquire the pose of the capsule robot in the stomach, and construct a localization model of the capsule robot based on the pose and magnetic sensor data, including: Select a point P in the space containing the magnetic shell, where the distance between P and the magnetic shell is greater than the diameter of the magnetic shell. Calculate the magnetic flux density at point P using the formula for the magnetic flux density of a magnetic dipole. Here, the length of the magnetic shell is L, the radius is r, and the surface of the magnetic shell has a uniform magnetization. The magnetic moment of a magnetic dipole is expressed as ,use To represent the standard direction vector of the magnetic field of the magnetic shell, Representing a spatial point Magnetic flux density at: （1） （2） （3） （4） in, This represents the relative permeability of a medium, where the medium refers to air. , Indicates the magnetic permeability in a vacuum; Indicates the position point from the magnetic shell to space point The vector, The magnetic field strength of a magnetic dipole is represented by the formula for the distance from the magnetic shell's position point to a point in space. , This represents the magnetic moment corresponding to the small circulation; When there are N sensors, the first... The position of each magnetic sensor in space is represented as follows: , No. The magnetic flux density measured by each magnetic sensor is: （5） According to formula (1), the magnetic induction intensity Decomposed into components in three orthogonal directions: （6） （7） （8） According to the direction vector of the magnetic shell get The cross product of both sides of formula (1) Dot product on both sides get And simplify to the linear form of the equation as follows ,in: in, , ,exist In the equation, and Based on known magnetic sensor data and the position parameters of the magnetic sensor The magnetic induction intensity values ​​measured by five magnetic sensors were calculated to obtain a matrix. and a vector ,get Solve for R and then solve the simultaneous equations. Substituting the values ​​into the elimination function yields the positioning parameters. The value of the positioning parameter, and the positioning parameter The value is used as the pose of the magnetic shell.

3. The method for detecting the gastric environment based on a capsule robot according to claim 2, characterized in that, Also includes: Acquire geomagnetic field data, and collect magnetic field data before and after the magnetic shell rotates; Based on the position coordinates and measurement data of a preset number of magnetic sensors, the position information of the magnetic shell is solved using the linear least squares method. ; Based on positioning parameters and location information Obtain attitude information The positioning model of the capsule robot is determined based on positioning parameters, attitude information, and position information.

4. The method for detecting the gastric environment based on a capsule robot according to claim 1, characterized in that, Acquire pH data from a pH sensor, and control the capsule endoscope to acquire multiple gastric images based on the pH data and the positioning model, including: The pH sensor probe uses a piston-driven mechanism to pressurize the internal reference electrolyte. The power required for piston propulsion is provided by a spring. After deformation, the inner hole of the silicone tube of the pH sensor probe becomes circular. Therefore, the outflow of electrolyte inside the silicone tube can be calculated using the Hagen-Poiseuille law. （9） Where Q represents traffic. It is the inner diameter of the silicone tube after deformation. It is the pressure difference per unit length. It is viscosity; the inflow pressure, outflow pressure, and flow rate of the silicone tube are obtained by analyzing the external flow rate of the internal reference electrolyte to obtain the value in formula (9). ,but The expression is: （10） in, It refers to the stiffness of the compression spring. It is the piston outer diameter. It is the initial volume of the internal reference solution. This refers to the volume of internal reference solution that flows out after the valve on the silicone tube has been open for a first preset time. The piston is powered by the spring.

5. The method for detecting the gastric environment based on a capsule robot according to claim 4, characterized in that, As the probe's usage time increases, the spring contraction decreases, and the pressure difference between the internal reference electrolyte and the external environment decreases. The integral form is: （11） in, It is the volume of the internal reference solution that flows out when the pressure of the internal reference solution drops to 5 kPa. It is the internal reference ratio to the initial pressure of the electrolyte. After the second preset duration of continuous operation, the internal pressure of the reference solution is: .

6. The method for detecting the gastric environment based on a capsule robot according to claim 1, characterized in that, Image preprocessing is performed on the multiple gastric images to obtain capsule endoscopy images, and image feature information of the capsule endoscopy images is extracted, including: Obtain a window centered on the edge pixels of the highlight region in the capsule endoscopy image, and calculate the sum of the ratios of the R and B channels of the known pixels in the RGB color space, where the sum of the ratios represents the proportion of the known pixels. The product of the gradient normal vector and the iso-illuminance line in the specular resolution of the Criminisi algorithm is used as the data item, where the data line indicates that there is a clear intersection boundary between the known area and the specular area; The confidence level is added to the weight of the data item to calculate the priority at the highlight edge, and the edge pixel with the highest priority is obtained. Variance adjustment of sample block size is used to repair highlight areas, and the best matching block is obtained by combining RGB color channel examples and pixel examples to repair the highest priority pixel window. The pixels in the best matching block are copied to the highlight region in the highest priority sample block, and the highlight region mask matrix is ​​updated until the mask matrix is ​​0. The boundary pixels of the highlight region are replaced with the average value of all pixels in the window centered on the boundary pixels to complete the highlight denoising of the capsule endoscopy image.

7. The method for detecting the gastric environment based on a capsule robot according to claim 6, characterized in that, The product of the gradient normal vector and the iso-illuminance line from the specular resolution in the Criminisi algorithm is used as the data item, including: Preset For capsule endoscopy images that need repair, The damaged area in the capsule endoscopy image. The damaged area in the capsule endoscopy image. The boundary between the known area and the damaged area. The pixel with the highest current priority value. For The central window, for The direction of the isoil lines at the point The Criminisi algorithm uses a gradient normal vector orthogonal to the boundary at each pixel. The priority calculation expression is: （12） in, for The confidence level of a point represents the window The expression for the proportion of pixels in a known region is: ; The larger the value, the more known area information it contains, and the more priority it will be for repair. for The data item of a point represents gradient normal vector of a point With isolux lines The expression for the product is: The larger the value, the more likely it is to indicate a clear boundary between the known and damaged areas, thus prioritizing repair. For window area, Normalization factor; Use window It is represented by the ratio of the R channel to the B channel of a known region of pixels. The higher the confidence level of a point, the greater the ratio of the R channel to the B channel. The more valid information a point has, the better the confidence level will be. The expression is: （13） in, This represents the ratio of the R channel to the B channel of pixel q. , and This represents the R and B channel values ​​in the RGB color space at point q; When the isolux line With gradient normal vector When vertical, data items If the value is 0, rewrite formula (12) as follows: ,in, These are the weighting coefficients.

8. The method for detecting the gastric environment based on a capsule robot according to claim 6, characterized in that, Capsule endoscopy images are preprocessed in HSV color space, and the images are decomposed from color space into hue, saturation, and brightness channels to separate color and brightness information, including: The extracted luminance channel image is subjected to adaptive CLAHE processing, and the luminance channel image is divided into multiple sub-regions, where each sub-region is of equal size, continuous, and non-overlapping. The histogram and required cropping value for each sub-region are calculated, and the corresponding expressions are as follows: （14） Where N represents the number of pixels in the sub-region, and M represents the number of gray levels contained in the corresponding sub-region. The cropping factor is calculated adaptively based on the weighted average of the image's grayscale mean and standard deviation. The relationship between gray values ​​and their corresponding cumulative distribution function (CDF) is fitted using a linear interpolation algorithm as follows: ,in, The calculation expression is , Let n represent the number of pixels with a grayscale value of n, and T represent the total number of pixels in the image. The calculation expression is: （15） in, For coefficient factors, The average gray level of the image pixels. The standard deviation of the image pixels is used to plot the gray-level histogram of the sub-region using the calculated cropping value. Perform cropping and calculate the total number of pixels exceeding the cropping value. The corresponding expression is: （16） The portion of pixels exceeding the clipping value is evenly distributed across all gray levels, with the number of pixels evenly distributed across each gray level set to [value missing]. The corresponding expression is Histogram after redistribution The calculation expression is: （17） The new histogram of each sub-region is equalized sequentially, and the final image is obtained through linear interpolation.

9. The method for detecting the gastric environment based on a capsule robot according to claim 1, characterized in that, The image feature information is input into a trained deep neural network to obtain image classification results. Based on the image classification results and pH data, the gastric environment detection results are determined, including: The deep neural network employs a Transform structure. The multi-head attention mechanism maps input data to multiple sub-regions, enabling the model to learn various feature representations of the input data from different sub-regions. In each attention head, the model calculates attention weights based on the similarity between query Q, key K, and value V, and uses these weights to perform a weighted average of the similarity values ​​to capture local and global feature information. The expression for the self-attention mechanism is: （18） in, , , These represent the query, key, and value, respectively. The key dimension is scaled by the dot product of the query and the key, and the attention weights obtained by normalization through the Softmax function are applied to the value V to obtain the weighted output result. If there are h attention heads, the output of the multi-attention mechanism is: （19） in, It is the output of the i-th head. It is the linear transformation matrix of the output layer; A feedforward neural network (FNN) is used to further perform a nonlinear transformation on the output of the attention layer. The FNN consists of two fully connected layers and an activation function. The preset input is U, and the output of the FNN is expressed as: （20） in, and It is a weight matrix. and It is the bias term, and ReLU is a non-linear activation function. ; A weighted average frame-level prediction aggregation function is used to calculate the final classification prediction by averaging the classification results of each frame. For each class j, the prediction score for each frame is calculated first. And calculate the corresponding weights based on the score of each frame. Weight This represents the contribution of the current frame to the final classification result. The expression for the weighted average aggregation function is: （21） in, For the final weighted classification result, This represents the weight of frame t in category j. Let represent the prediction score for frame t, and the weight expression for each frame is: , .

10. A gastric environment detection system based on a capsule robot, characterized in that, The method for detecting the gastric environment based on a capsule robot as described in any one of claims 1-9 includes: The model building module is used to obtain the pose of the capsule robot in the stomach and build a localization model of the capsule robot based on the pose and magnetic sensor data, wherein the pose includes the position information and attitude information of the magnetic shell; The image acquisition module is used to acquire pH data collected by the pH sensor, and control the capsule endoscope to acquire multiple stomach images based on the pH data and the positioning model; The feature extraction module is used to preprocess the multiple gastric images to obtain capsule endoscopy images and extract the image feature information of the capsule endoscopy images; The classification and detection module is used to input the image feature information into a trained deep neural network to obtain image classification results, and to determine the gastric environment detection results based on the image classification results and pH data.

Images