A classification system for intestinal diseases based on multimodal data

Abstract

The present invention relates to a multimodal data-based intestinal disease classification system, addressing the technical field of intestinal disease classification and addressing the inaccurate intestinal disease classification issues encountered in existing technologies. The system comprises a bowel sound acquisition unit, which uses a distributed fiber optic sensor to non-contactly acquire bowel sound signals from patients; an image acquisition unit, which acquires ultrasound images of the patient's abdomen; and an intestinal disease classification module, which inputs the patient's bowel sound signals and ultrasound images into a trained multi-channel classification model to obtain a classification result for the patient's intestinal disease. This improves the accuracy of intestinal disease detection.

Description

Technical Field

[0001] The present invention relates to the technical field of intestinal disease classification, and in particular to an intestinal disease classification system based on multimodal data. Background Art

[0002] Intestinal diseases, such as inflammatory bowel disease (IBD), irritable bowel syndrome (IBS), and intestinal obstruction, are associated with significant morbidity and mortality worldwide. Early diagnosis and intervention are crucial to improving patient outcomes.

[0003] Bowel sounds are the sounds produced by the friction between intestinal contents (solid, gas or liquid) and the intestines, or between intestinal contents, under the premise of intestinal peristalsis. They can reflect the state of intestinal movement and are related to clinical applications such as the physiological state of the human body and disease diagnosis.

[0004] Abdominal ultrasound images have the advantages of being non-invasive, real-time, and dynamic in intestinal disease monitoring. They can provide information on intestinal structure, function, blood flow, and other aspects, and are an important tool for the diagnosis, treatment, and follow-up of intestinal diseases.

[0005] Traditional methods rely primarily on single-modality data, such as clinical symptoms, stool testing, endoscopy, or imaging. Each of these methods has its own advantages and disadvantages, including the subjective nature of clinical symptoms, the low specificity of stool testing, the invasiveness and high cost of endoscopy, and the lack of sensitivity of imaging for early-stage lesions. Single-modality data cannot fully reflect the complexity and heterogeneity of intestinal diseases, easily leading to misdiagnosis and missed diagnoses. Summary of the Invention

[0006] In view of the above analysis, an embodiment of the present invention aims to provide an intestinal disease classification system based on multimodal data to solve the problem of inaccurate classification of existing intestinal diseases.

[0007] In one aspect, an embodiment of the present invention provides an intestinal disease classification system based on multimodal data, comprising:

[0008] A bowel sound acquisition unit, which is used to acquire the patient's bowel sound signals in a non-contact manner based on a distributed optical fiber sensor;

[0009] An image acquisition unit, configured to acquire an abdominal ultrasound image of the patient;

[0010] The intestinal disease classification module is used to input the patient's bowel sound signals and abdominal ultrasound images into the trained multi-channel classification model to obtain the patient's intestinal disease classification results.

[0011] Based on the further improvement of the above technical solution, the multi-channel classification model includes:

[0012] A bowel sound feature extraction module is used to extract features of each signal frame from the input bowel sound signal;

[0013] An image feature extraction module, used to extract features of an input abdominal ultrasound image;

[0014] A fusion module is used to fuse the features of each signal frame with the features of the abdominal ultrasound image based on a graph network to obtain a fusion feature;

[0015] The classification module is used to classify intestinal diseases based on fusion features and obtain the type of intestinal disease.

[0016] Based on a further improvement of the above technical solution, the graph network includes a first graph unit and a second graph unit; the graph network is constructed in the following manner:

[0017] The abdominal ultrasound image is divided into multiple regions, each region corresponds to a first node, and edges between the first nodes are constructed based on the similarity between the first nodes to obtain a first graph unit; each signal frame is regarded as a second node, and edges between the second nodes are constructed based on the similarity between the second nodes to obtain a second graph unit;

[0018] The fusion module fuses the features of each signal frame with the features of the abdominal ultrasound image to obtain fusion features in the following manner:

[0019] Aggregating the first graph units multiple times based on the adjacency relationship between the first nodes to obtain the output feature of each first node;

[0020] Aggregating the second graph units multiple times based on the adjacency relationship between the second nodes to obtain the output feature of each second node;

[0021] The output features of all the first nodes and the second nodes are fused to obtain the fused features.

[0022] Based on the further improvement of the above technical solution, the following formula is used to calculate the characteristics of each first node after each aggregation:

[0023] ; ;

[0024] in, represents the feature of the first node i after the t-th aggregation, and Indicates the parameters of the t-th aggregation, represents the feature of the first node i after the t-1th aggregation, represents the aggregated features of the adjacent nodes of the i-th first node, represents the sigmoid function, represents the concatenation function, represents the mean function, Represents the set of adjacent nodes of the i-th first node.

[0025] Based on the further improvement of the above technical solution, the following formula is used to calculate the training loss of the multi-channel classification model:

[0026] ;

[0027] in, represents the classification loss, represents the node distance loss, and Represents weight.

[0028] Based on the further improvement of the above technical solution, the following formula is used to calculate the node distance loss:

[0029] ;

[0030] in, represents the output feature of the i-th node, Represents the output feature of the jth node, the jth node is the adjacent node of the i-th node, Represents the output feature of the u-th node, the u-th node is the comparison node of the i-th node, m represents the number of samples of the comparison node, and the superscript T represents transposition. Represents the sigmoid function, and n represents the number of nodes.

[0031] Based on the further improvement of the above technical solution, the comparison node of the i-th node is sampled in the following way:

[0032] Calculate the adjacency overlap between each non-adjacent node of the i-th node and the i-th node;

[0033] Select m nodes with larger adjacency overlap as comparison nodes for the i-th node.

[0034] Based on the further improvement of the above technical solution, the following formula is used to calculate the adjacency overlap between each non-adjacent node and the i-th node:

[0035] ;

[0036] in, represents the degree of the vth non-adjacent node, represents the set of adjacent nodes of the i-th node, represents the set of adjacent nodes of the vth non-adjacent node, It represents the adjacency overlap between the vth non-adjacent node and the i-th node.

[0037] Based on the further improvement of the above technical solution, the bowel sound collection unit includes:

[0038] The signal acquisition device includes a laser, a distributed optical fiber sensor, and a collector; the laser is used to periodically emit laser light to the distributed optical fiber sensor; the collector is used to collect the signal returned by the optical fiber sensor and send it to the bowel sound solution module;

[0039] The bowel sound calculation module is used to calculate the collected signal based on the phase demodulation algorithm and extract the bowel sound signal.

[0040] Based on a further improvement of the above technical solution, the distributed fiber optic sensor includes a test fiber, a reference fiber, a first coupler and a circulator; the first coupler is connected to the laser and is used to split the light source into two light paths, one of which is injected into the reference fiber and the other is injected into the test fiber through the circulator;

[0041] The collector includes a second coupler and a photodetector. The second coupler is connected to the reference optical fiber and the circulator, and is used to couple the signals returned by the reference optical fiber and the test optical fiber and send them to the photodetector; the photodetector converts the optical signal into an electrical signal and sends it to the bowel sound solution module.

[0042] Compared with the existing technology, the present invention uses a bowel sound acquisition unit to collect the patient's bowel sound signals in a non-contact manner, thereby eliminating interference from human activities, friction, etc., and the extracted signals are more accurate. By collecting abdominal ultrasound images, the bowel sound signals and abdominal ultrasound images are input into a trained multi-channel classification model, thereby combining multimodal data to improve the accuracy of intestinal disease detection and improve the level of early screening and diagnosis of intestinal diseases.

[0043] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Like reference symbols denote like components throughout the accompanying drawings.

[0045] Figure 1 This is a block diagram of an intestinal disease classification system based on multimodal data according to an embodiment of the present invention;

[0046] Figure 2 is a block diagram of a bowel sound acquisition unit according to an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a portion of the structure of a signal acquisition device implemented in the present invention;

[0048] Figure 4 A schematic diagram of the signal acquisition device implemented in the present invention;

[0049] Reference numerals:

[0050] 1-outer shell of cylindrical shell, 2-optical fiber, 3-movable bed, 4-sound insulation material, 5-bell mouth, 6-inner shell of cylindrical shell, 7-optical fiber section, 8-vibration-absorbing cavity. DETAILED DESCRIPTION

[0051] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0052] A specific embodiment of the present invention discloses a system for classifying intestinal diseases based on multimodal data, such as Figure 1 Shown, including:

[0053] A bowel sound acquisition unit, which is used to acquire the patient's bowel sound signals in a non-contact manner based on a distributed optical fiber sensor;

[0054] An image acquisition unit, configured to acquire an abdominal ultrasound image of the patient;

[0055] The intestinal disease classification module is used to input the patient's bowel sound signals and abdominal ultrasound images into the trained multi-channel classification model to obtain the patient's intestinal disease classification results.

[0056] Compared with the existing technology, the intestinal disease classification system based on multimodal data provided in this embodiment uses a bowel sound acquisition unit to collect the patient's bowel sound signals in a non-contact manner, thereby eliminating interference from human activities, friction, etc., and the extracted signals are more accurate. By collecting abdominal ultrasound images, the bowel sound signals and abdominal ultrasound images are input into a trained multi-channel classification model, thereby combining multimodal data to improve the accuracy of intestinal disease detection and improve the level of early screening and diagnosis of intestinal diseases.

[0057] Existing bowel sound acquisition devices typically use traditional acoustic sensors, such as coil microphones, and attach one or more sensors near the abdominal cavity via a waistband to monitor bowel sounds. The waistband-mounted acoustic sensor array is in contact with the human body, generating significant frictional noise during human activity, which can significantly interfere with bowel sound signals. Furthermore, this noise significantly increases data volume and reduces data quality, complicating subsequent bowel sound identification. The combination of a waistband-mounted sensor array and a traditional acoustic probe limits the number of sensors, resulting in insufficient accuracy for time-delay acoustic localization. Because the wavelength of sound waves is much larger than the size of the abdominal cavity and the reflection and superposition of sound waves in human tissue are significant, traditional methods of bowel sound location accuracy are poor. Furthermore, human body variability and movement during wear can lead to inaccurate initial sensor positioning and displacement, resulting in ineffective bowel sound localization and degraded data quality. Therefore, current bowel sound monitoring and identification suffers from numerous deficiencies in the basic data collection phase, significantly increasing the difficulty of subsequent bowel sound identification.

[0058] Optical fiber is a new communication medium. As a signal transmission medium, it can transmit signals over distances ranging from a few meters to tens or even hundreds of kilometers. In many fields, it has gradually replaced traditional wireless communication technologies. When light travels through an optical fiber, photons collide with molecules and scatter. Distributed optical fiber sensing achieves this by detecting the one-to-one correspondence between these scattered signals and external physical quantities.

[0059] Therefore, the bowel sound acquisition unit of the present invention collects the patient's bowel sound signals in a non-contact manner based on a distributed optical fiber sensor. Any position on the test optical fiber of the distributed optical fiber sensor can be used as a sensor, so the number of measurement points is much higher than that of the traditional sensor acquisition method. In addition, the distributed optical fiber sensor does not need to contact the individual being measured, so the positioning is more accurate and the collected signal quality is higher, thereby improving the accuracy of detection.

[0060] Specifically, such as Figure 2 As shown, the bowel sound collection unit includes:

[0061] The signal acquisition device includes a laser, a distributed optical fiber sensor, and a collector; the laser is used to periodically emit laser light to the distributed optical fiber sensor; the collector is used to collect the signal returned by the optical fiber sensor and send it to the bowel sound solution module;

[0062] The bowel sound calculation module is used to calculate the collected signal based on the phase demodulation algorithm and extract the bowel sound signal.

[0063] During implementation, the light source of the laser is a tunable laser source, and the laser emits a linear frequency sweep signal in each emission cycle.

[0064] Specifically, the distributed fiber optic sensor includes a test fiber, a reference fiber, a first coupler, and a circulator; the first coupler is connected to the laser and is used to split the light source into two light paths, one of which is injected into the reference fiber and the other is injected into the test fiber through the circulator;

[0065] The collector includes a second coupler and a photodetector. The second coupler is connected to the reference optical fiber and the circulator, and is used to couple the signals returned by the reference optical fiber and the test optical fiber and send them to the photodetector; the photodetector converts the optical signal into an electrical signal and sends it to the bowel sound solution module.

[0066] The light source is divided into two beams through the first coupler. One beam enters the reference optical fiber; the other beam enters the circulator through port 1 and exits through port 2 to enter the test optical fiber. Part of the light in the test optical fiber will be reflected back, and the reflected light enters the second coupler through port 3 of the circulator. The two light beams have different optical paths and optical frequencies, so a beat frequency signal is generated at the second coupler. The photodetector collects the beat frequency signal, converts it into an electrical signal, and sends it to the bowel sound resolution module.

[0067] During implementation, in order to facilitate signal collection, the signal acquisition device also includes a cylindrical shell; there is an annular groove on the inner wall of the cylindrical shell; the test optical fiber of the distributed optical fiber sensor is wound on the inner wall of the cylindrical shell, and the test optical fiber of the distributed optical fiber sensor is embedded in the groove on the inner wall of the cylindrical shell.

[0068] During implementation, annular grooves are arranged at equal intervals on the inner wall of the cylindrical shell.

[0069] When implementing, if Figure 3 As shown, the cylindrical shell includes an outer shell and an inner wall, and the space between the outer shell and the inner wall is filled with sound insulation material. The test optical fiber is wrapped in a shock-absorbing cavity, thereby reducing the influence of the external environment on the sensor. In order to improve the detection sensitivity, the shock-absorbing cavity on the measuring surface of the optical fiber adopts a thin shell, or no shell, so that the optical fiber can more accurately detect physiological sound signals. By providing a groove on the inner wall, the test optical fiber is embedded in the groove. The cross-section of the groove is trumpet-shaped, forming a trumpet mouth. The configuration of the trumpet mouth allows the sensor to monitor a specific small area of ​​the abdomen in a targeted manner, thereby avoiding environmental interference and vibration interference from other parts (such as breathing, heartbeat, etc.), and focusing on the sound signal collection of the small area corresponding to the trumpet mouth, achieving physical noise reduction, which is more suitable for receiving bowel sound signals.

[0070] During the implementation, the individual to be tested lies on the mobile bed, and the mobile bed is pushed into the cylindrical shell for easy testing. Figure 4 The area corresponding to the abdomen on the bed is hollowed out so that the patient's back directly faces the lower sensor array to avoid affecting the transmission of bowel sound signals.

[0071] During implementation, the signal acquisition device collects signals for 2 to 4 minutes, and sends the collected signals to the bowel sound solution module for settlement to obtain bowel sound signals.

[0072] During implementation, an ultrasound image of the patient's abdomen is acquired by an image acquisition unit.

[0073] After receiving the bowel sound signal and abdominal ultrasound image, the intestinal disease classification module inputs the trained multi-channel classification model to predict the patient's intestinal disease type.

[0074] Specifically, the multi-channel classification model includes:

[0075] A bowel sound feature extraction module is used to extract features of each signal frame from the input bowel sound signal;

[0076] An image feature extraction module, used to extract features of an input abdominal ultrasound image;

[0077] A fusion module is used to fuse the features of each signal frame and then the features of the abdominal ultrasound image based on the graph network to obtain a fusion feature;

[0078] The classification module is used to classify intestinal diseases based on fusion features and obtain the type of intestinal disease.

[0079] During implementation, the bowel sound feature extraction module and the image feature extraction module are used to extract sound signal features and image features respectively.

[0080] Specifically, the bowel sound feature extraction module includes:

[0081] A convolutional encoding module is used to perform convolution processing on the bowel sound signal to obtain the initial features of each signal frame; the initial features of the signal frame are randomly masked and input into the BERT encoding module;

[0082] The BERT encoding module outputs the features of each signal frame based on the initial feature sequence of the randomly masked signal frame. In implementation, the sound signal is a continuous time series signal. To more deeply extract signal features and improve detection accuracy, the bowel sound feature extraction module convolves the bowel sound signal with the convolutional encoding module, dividing the bowel sound signal into multiple consecutive signal frames. For example, convolution of a 16kHz sampling rate bowel sound signal produces a feature sequence with a 20ms frame rate. This means that every 20ms is considered a signal frame, and the initial features of each signal frame are obtained. In implementation, the convolutional encoding module can utilize an existing convolutional neural network module.

[0083] When implemented, the BERT encoding module adopts the existing BERT encoder structure.

[0084] In implementation, the acoustic features of the original signal frame (e.g., MFCC) can be used as input to the clustering module. After clustering, the centroid or center point of each cluster type is used as the label of the cluster type.

[0085] During implementation, the initial features of the signal frame sequence are randomly masked and input into the BERT encoding module, and the BERT encoding module outputs the features of each signal frame based on the initial feature sequence of the signal frame after random masking.

[0086] During implementation, the image feature extraction module may adopt an existing image feature extraction network structure, such as an encoder-decoder symmetric network (such as a Unet network structure), and the decoder outputs a feature map of the abdominal ultrasound image.

[0087] During implementation, the fusion module fuses the features of the characteristic abdominal ultrasound image of each signal frame based on the graph network to obtain the fused features.

[0088] During implementation, the graph network includes a first graph unit and a second graph unit; and the graph network is constructed in the following manner:

[0089] The abdominal ultrasound image is divided into multiple regions, each region corresponds to a first node, and edges between the first nodes are constructed based on the similarity between the first nodes to obtain a first graph unit; each signal frame is regarded as a second node, and edges between the second nodes are constructed based on the similarity between the second nodes to obtain a second graph unit.

[0090] During implementation, the abdominal ultrasound image is segmented into multiple regions, each corresponding to a first node. Fixed-size segmentation or superpixel segmentation can be used. For each region, the features of the corresponding region in the feature map of the abdominal ultrasound image are the features of the 0th aggregation of the first node corresponding to that region.

[0091] By calculating the similarity between the first nodes, an edge between the first nodes is constructed. For example, an edge is constructed between two first nodes whose similarity exceeds a preset threshold, thereby constructing a first graph unit.

[0092] Similarly, each signal frame is treated as a second node. The similarity of the features of two signal frames is calculated, and an edge is constructed between two second nodes whose similarity exceeds a preset threshold. This constructs the second graph unit. The features extracted by the bowel sound feature extraction module for each signal frame are the zeroth-aggregated features of each corresponding second node.

[0093] The fusion module fuses the features of each signal frame with the features of the abdominal ultrasound image to obtain fused features in the following way:

[0094] Aggregating the first graph units multiple times based on the adjacency relationship between the first nodes to obtain the output feature of each first node;

[0095] Aggregating the second graph units multiple times based on the adjacency relationship between the second nodes to obtain the output feature of each second node;

[0096] The output features of all the first nodes and the second nodes are fused to obtain the fused features.

[0097] The following description is made by taking the multiple aggregations of the first graph unit as an example.

[0098] Specifically, the following formula is used to calculate the features of each first node after each aggregation:

[0099] ; ;

[0100] in, represents the feature of the first node i after the t-th aggregation, and Indicates the parameters of the t-th aggregation, represents the feature of the first node i after the t-1th aggregation, represents the aggregated features of the adjacent nodes of the i-th first node, represents the sigmoid function, represents the concatenation function, represents the mean function, Represents the set of adjacent nodes of the i-th first node.

[0101] During implementation, the adjacent nodes of the i-th first node are the K-hop adjacent nodes of the i-th first node in the first graph unit.

[0102] Likewise, the adjacent nodes of the second node are the K-hop adjacent nodes of the second node in the second graph unit.

[0103] During implementation, at the t-th aggregation, the features of the i-th first node after the t-1 aggregation and the features of its adjacent nodes after the t-1 aggregation are averaged. The average features are then transformed nonlinearly to obtain the aggregated features of the adjacent nodes. The aggregated features are then concatenated with the features of the i-th first node after the t-1 aggregation to obtain the features of the i-th first node after the t-1 aggregation. The features after the final aggregation become the output features of the i-th first node.

[0104] It should be noted that after each aggregation, the edges between nodes are re-established based on the feature similarity between the nodes. By performing multiple aggregations on the first graph units based on the adjacency relationship between the first nodes, the features of the target nodes are updated in combination with the adjacent node information. The node features change dynamically according to the adjacency relationship, thereby extracting more accurate features.

[0105] The fusion module performs the same process of aggregating the second graph units multiple times based on the adjacency relationship between the second nodes to obtain the output features of each second node. The output features of all first and second nodes are fused to obtain the fused features. During implementation, the output features of all first and second nodes can be concatenated to obtain the fused features.

[0106] Then, the intestinal diseases are classified through the classification module. During implementation, the classification module can adopt the existing classifier structure.

[0107] During implementation, a training sample set is constructed by collecting bowel sound signals and abdominal ultrasound images of different patients and corresponding intestinal disease type labels; and the constructed multi-channel classification model is trained based on the constructed training sample set.

[0108] Specifically, the following formula is used to calculate the training loss of the multi-channel classification model:

[0109] ;

[0110] in, represents the classification loss, represents the node distance loss, and Represents weight.

[0111] In implementation, the classification loss can be calculated using a cross-entropy loss function based on the labels and the model's predictions.

[0112] Specifically, the node distance loss is calculated using the following formula:

[0113] ;

[0114] in, represents the output feature of the i-th node, Represents the output feature of the jth node, the jth node is the adjacent node of the i-th node, Represents the output feature of the u-th node, the u-th node is the comparison node of the i-th node, m represents the number of samples of the comparison node, and the superscript T represents transposition. Represents the sigmoid function, and n represents the number of nodes, that is, the total number of first nodes and second nodes.

[0115] During implementation, adjacent nodes should have similar features, and the features of non-adjacent nodes should be different. Since there are a large number of non-adjacent nodes, some nodes are sampled from them as comparison nodes. Through distance loss, adjacent nodes are made to have similar vector representations, while the feature representations of separated nodes (comparison nodes) are made as distinguishable as possible.

[0116] During implementation, a node is randomly selected from the adjacent nodes of the i-th node as a positive sample, and its output feature is , sample m nodes from the non-adjacent nodes of the i-th node as comparison nodes to calculate the loss.

[0117] During implementation, the comparison node of the i-th node is sampled in the following way:

[0118] Calculate the adjacency overlap between each non-adjacent node of the i-th node and the i-th node;

[0119] Select m nodes with larger adjacency overlap as comparison nodes for the i-th node.

[0120] During implementation, nodes with similar structures to the i-th node are selected through the adjacency overlap. The higher the adjacency overlap, the more similar the two nodes are in the graph structure and the more difficult they are to distinguish. Therefore, by comparing common adjacent nodes, the classification accuracy of the model can be improved.

[0121] Specifically, the following formula is used to calculate the adjacency overlap between each non-adjacent node of the i-th node and the i-th node:

[0122] ;

[0123] in, represents the degree of the vth non-adjacent node, represents the set of adjacent nodes of the i-th node, represents the set of adjacent nodes of the vth non-adjacent node, It represents the adjacency overlap between the vth non-adjacent node and the i-th node.

[0124] By using the degree of the vth non-adjacent node as a weight, valid non-adjacent nodes are further selected as comparison nodes.

[0125] Improve the classification accuracy of the model by considering classification loss and clustering loss.

[0126] The parameters of the multi-channel classification model are updated inversely based on the loss. When the training stop condition is met, the training is stopped to obtain a trained multi-channel classification model. The training stop condition can be, for example, reaching a preset loss accuracy or the number of training times.

[0127] By inputting the bowel sound signals and abdominal ultrasound images of the patient to be predicted into the trained multi-channel classification model, the patient's intestinal disease classification results can be obtained quickly and accurately, thereby improving the accuracy of classification.

[0128] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0129] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A system for classifying intestinal diseases based on multimodal data, characterized in that: include: A bowel sound acquisition unit, which is used to acquire the patient's bowel sound signals in a non-contact manner based on a distributed optical fiber sensor; An image acquisition unit, configured to acquire an abdominal ultrasound image of the patient; The intestinal disease classification module is used to input the patient's bowel sound signals and abdominal ultrasound images into the trained multi-channel classification model to obtain the patient's intestinal disease classification results; The multi-channel classification model includes: A bowel sound feature extraction module is used to extract features of each signal frame from the input bowel sound signal; An image feature extraction module, used to extract features of an input abdominal ultrasound image; A fusion module is used to fuse the features of each signal frame with the features of the abdominal ultrasound image based on a graph network to obtain a fusion feature; The classification module is used to classify intestinal diseases based on fusion features and obtain the type of intestinal diseases; The graph network includes a first graph unit and a second graph unit; the graph network is constructed in the following manner: The abdominal ultrasound image is divided into multiple regions, each region corresponds to a first node, and edges between the first nodes are constructed based on the similarity between the first nodes to obtain a first graph unit; each signal frame is regarded as a second node, and edges between the second nodes are constructed based on the similarity between the second nodes to obtain a second graph unit; The fusion module fuses the features of each signal frame with the features of the abdominal ultrasound image to obtain fusion features in the following manner: Aggregating the first graph units multiple times based on the adjacency relationship between the first nodes to obtain the output feature of each first node; Aggregating the second graph units multiple times based on the adjacency relationship between the second nodes to obtain the output feature of each second node; The output features of all the first nodes and the second nodes are fused to obtain the fused features.

2. The intestinal disease classification system based on multimodal data according to claim 1, characterized in that: The following formula is used to calculate the characteristics of each first node after each aggregation: ； ； in, represents the feature of the first node i after the t-th aggregation, and Indicates the parameters of the t-th aggregation, represents the feature of the first node i after the t-1th aggregation, represents the aggregated features of the adjacent nodes of the i-th first node, represents the sigmoid function, represents the concatenation function, represents the mean function, Represents the set of adjacent nodes of the i-th first node.

3. The intestinal disease classification system based on multimodal data according to claim 1, characterized in that: The training loss of the multi-channel classification model is calculated using the following formula: ； in, represents the classification loss, represents the node distance loss, and Represents weight.

4. The intestinal disease classification system based on multimodal data according to claim 3, characterized in that: The node distance loss is calculated using the following formula: ； in, represents the output feature of the i-th node, Represents the output feature of the jth node, the jth node is the adjacent node of the i-th node, Represents the output feature of the u-th node, the u-th node is the comparison node of the i-th node, m represents the number of samples of the comparison node, and the superscript T represents transposition. Represents the sigmoid function, and n represents the number of nodes.

5. The intestinal disease classification system based on multimodal data according to claim 4, characterized in that: The comparison node of the i-th node is sampled in the following way: Calculate the adjacency overlap between each non-adjacent node of the i-th node and the i-th node; Select m nodes with larger adjacency overlap as comparison nodes for the i-th node.

6. The intestinal disease classification system based on multimodal data according to claim 5, characterized in that: The following formula is used to calculate the adjacency overlap between each non-adjacent node and the i-th node: ; in, represents the degree of the vth non-adjacent node, represents the set of adjacent nodes of the i-th node, represents the set of adjacent nodes of the vth non-adjacent node, It represents the adjacency overlap between the vth non-adjacent node and the i-th node.

7. The intestinal disease classification system based on multimodal data according to claim 1, characterized in that: The bowel sound collection unit includes: The signal acquisition device includes a laser, a distributed optical fiber sensor, and a collector; the laser is used to periodically emit laser light to the distributed optical fiber sensor; the collector is used to collect the signal returned by the optical fiber sensor and send it to the bowel sound solution module; The bowel sound calculation module is used to calculate the collected signal based on the phase demodulation algorithm and extract the bowel sound signal.

8. The intestinal disease classification system based on multimodal data according to claim 7, characterized in that: The distributed fiber optic sensor includes a test fiber, a reference fiber, a first coupler, and a circulator; the first coupler is connected to a laser and is used to split the light source into two light paths, one of which is injected into the reference fiber and the other is injected into the test fiber through the circulator; The collector includes a second coupler and a photodetector, wherein the second coupler is connected to the reference optical fiber and the circulator, and is used to couple the signals returned by the reference optical fiber and the test optical fiber and then send them to the photodetector; The photoelectric detector converts the optical signal into an electrical signal and sends it to the bowel sound calculation module.

Images