Drainage fluid monitoring method and system based on entity reasoning hybrid expert module

Abstract

The drainage fluid monitoring method of the present application comprises the steps of: acquiring a drainage fluid image I to be identified, extracting its spatial visual feature sequence V using a pre-trained backbone network; constructing an entity reasoning hybrid expert module, inputting the spatial visual feature sequence V into the entity reasoning hybrid expert module, and extracting each attribute representation; constructing a structured entity prototype and initializing it to generate each semantic prototype; coupling the spatial visual feature sequence, the attribute representation, and the semantic prototype, and calculating the corresponding fusion weight; fusing the fusion weight and the attribute representation to generate a visual representation, decoding the visual representation Z to generate a drainage clinical report; the present application can effectively assist the automatic monitoring and complication warning of postoperative drainage.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence technology, and in particular relates to a method and system for monitoring drainage fluid based on entity reasoning hybrid expert modules. Background Technology

[0002] Perioperative drainage monitoring is a key indicator reflecting a patient's surgical condition, particularly crucial for the early detection of complications in hepatobiliary surgery. However, current clinical surgical drainage monitoring still heavily relies on manual examination and handwritten records. This not only depends heavily on manpower but is also prone to errors. Furthermore, the discontinuity of monitoring often leads to delayed detection of complications, impacting patient recovery. While advancements have been made in automated radiology report generation technology, the field of intelligent drainage monitoring still faces the following challenges: (1) Lack of large-scale, clinically labeled drainage fluid image-report datasets; (2) Since it differs from the imaging logic of internal organs, it is necessary to make inferences by comprehensively considering the fluid appearance, local morphology and overall context of the drainage fluid; (3) Existing models rely heavily on global semantics, making it difficult to accurately locate subtle clinical attributes.

[0003] To address the issues of strong heterogeneity in drainage fluid properties and difficulty in aligning visual features with clinical semantics, there is an urgent need to develop a method for intelligent monitoring of drainage fluid in order to generate accurate clinical reports on drainage fluid. Summary of the Invention

[0004] To address the shortcomings of existing drainage fluid monitoring methods, such as strong heterogeneity in properties, difficulty in aligning visual features with clinical semantics, and the problems of reliance on heavy human intervention leading to monitoring errors and discontinuities in observation, this invention provides a drainage fluid monitoring method and system based on an entity-based reasoning hybrid expert module. To achieve the above objectives, this invention utilizes the following technical solutions: A method for monitoring drainage fluid based on an entity-reasoning hybrid expert module, characterized by the following steps: Step S1: Obtain the drainage fluid image I to be identified, and extract the spatial visual feature sequence V of the drainage fluid image I using a pre-trained backbone network; Step S2: Construct an entity reasoning hybrid expert module, which includes a global context expert module, a fluid dynamics expert module, and a local morphology expert module; input the spatial visual feature sequence V into the entity reasoning hybrid expert module, and extract the attribute representations corresponding to the entity reasoning hybrid expert module. ; Step S3: Construct and initialize a structured entity prototype to generate a semantic prototype corresponding to the entity reasoning hybrid expert module. ; Step S4: Sequence of spatial visual features Attribute representation With semantic prototype Couple the components and calculate the fusion weights corresponding to the hybrid expert module for entity reasoning. ; Step S5: Combine the fusion weights corresponding to the entity reasoning hybrid expert module. and attribute representation By fusing the data, a visual representation can be generated. The visual representation Z is decoded using a decoder to generate a clinical report on drainage.

[0005] Preferably, the implementation of step S1 includes: Step S11: Acquire a visible light image of the drainage bag using an image acquisition device to obtain the drainage fluid image I to be identified; Step S12: Process the drainage fluid image I into a format that meets the input requirements of the pre-trained backbone network to obtain the adapted drainage fluid image I'; Step S13: Input the adapted drainage fluid image I' into the pre-trained backbone network for forward computation to obtain the spatial feature response map corresponding to the drainage fluid image I'; Step S14: Expand the spatial feature response map in terms of spatial dimensions to form a spatial visual feature sequence V; The pre-trained backbone network is a ResNet-101 network.

[0006] Preferably, the processing method in step S12 includes one or more of size normalization / scaling, pixel value normalization, and channel format alignment; The implementation method of spatial dimension unfolding in step S14 is as follows: flatten or rearrange according to spatial position, and record the feature vector corresponding to each spatial position as a visual word.

[0007] Preferably, the attribute characterization in step S2 The extraction methods are as follows: The spatial visual feature sequence V is input into the overall context expert module to obtain the context-enhanced attribute representation. : Formula 1 Mamba represents a deep learning network based on a state-space model that captures long-range global contextual information. The spatial visual feature sequence V is input into the fluid dynamics expert module. A region response is generated through nonlinear activation, and the spatial visual feature sequence V is subjected to element-wise modulation and projection transformation to obtain the attribute representation. : Formula 2 In the formula, C represents the number of channels of V. express Visual features of the c-th channel, c Here, c represents the classifier weights, and c is a non-zero natural number. The spatial visual feature sequence V is input into the local morphology expert module. Local features under different receptive fields are obtained through multi-branch convolutional filtering, and spatial attention weights are generated for each branch for weighted fusion to obtain attribute representations. ; Formula 3 Formula 4 Formula 5 In the formula, k represents the size of three different convolution kernels: 3, 5, and 7. Representatives utilize Convolution of convolution kernels; This represents intermediate variables in the calculation process. Represents average pooling. Represents max pooling. This represents the MLP projection layer, i.e., the multilayer perceptron module.

[0008] Preferably, the implementation of step S3 includes: Step S31: Based on the report generation requirements of the clinical monitoring task of drainage fluid, a labeling system for nine clinical attributes of drainage is pre-established; the nine clinical attributes of drainage include drainage source, drainage tube type, complication type, drainage fluid volume, drainage fluid clarity, drainage fluid color, deposits on the drainage bag wall, sediments in the drainage fluid, and sediment color; Step S32: For each clinical attribute of the patient's flow, based on the clinical report writing standards, medical terminology definitions, and the annotation content in the training data, organize multiple medical entity terms corresponding to the current clinical attribute of the patient's flow to form a corresponding attribute entity set; Step S33: Input the attribute entity sets corresponding to each clinical attribute of drainage into the PubMedBERT pre-trained model for encoding to obtain semantic embedding vectors corresponding to the medical entity terms in each attribute entity set. and form a structured entity prototype. ;in, Refers to the ordinal numbers of the nine clinical attributes of drainage. Refers to the first The number of entities possessed in each clinical attribute for drainage, n being [1, N]. i ]; Step S34: Evaluate the structured entity prototype using an attention aggregation generation method. Importance weights; Step S35: Based on the normalized weights obtained in step S34, adjust the semantic embedding vectors corresponding to each medical entity term. Perform weighted aggregation to generate an attribute semantic indicator vector corresponding to the current clinical attributes of the drainage. Step S36: Classify based on the differences in the semantic indicator vectors of the clinical attributes of drainage; Step S37: Based on the classification results, establish the correspondence between the attribute semantic indicator vector and the entity reasoning hybrid expert module, and generate the semantic prototype corresponding to the entity reasoning hybrid expert module. .

[0009] Preferably, the implementation of step S34 includes: This structured entity prototype Each semantic embedding vector Perform linear transformations on each vector to map them to a unified semantic representation space; then, convert the linearly transformed semantic embedding vectors... The semantic query information is correlated with the corresponding clinical attributes of the target audience to obtain the original score for each medical entity term. Then, each original score is normalized within the same entity semantic set to obtain the normalized weight for each medical entity term. The normalized weight is used to characterize each medical entity term within the current structured entity prototype. The relative contribution of [the entity / entity].

[0010] Preferably, the implementation of step S4 includes: Step S41: Perform global aggregation processing on the spatial visual feature sequence V to form an image-level semantic representation that represents the overall semantic information of the current input image; then combine the image-level semantic representation with the three semantic prototypes. Perform relevance calculations to obtain initial relevance scores corresponding to the three semantic prototypes; Step S42: Represent the attributes respectively The aggregation process is performed to obtain the corresponding overall feature representation, which is then compared with the corresponding semantic prototypes. Feature coupling is performed to construct a joint feature representation that simultaneously includes visual attribute information and structured semantic priors; then the joint feature representation after feature coupling is normalized and nonlinearly transformed, and input into a gating network for calculation to obtain fine-grained relevance scores corresponding to the hybrid expert module for entity reasoning. Step S43: The initial relevance score obtained in step S41 and the fine-grained relevance score obtained in step S42 are weighted and fused to obtain the fusion score corresponding to the entity reasoning hybrid expert module respectively; then the fusion score is normalized by the Softmax function to obtain the fusion weight corresponding to each entity reasoning hybrid expert module. .

[0011] Preferably, the correlation calculation in step S41 is performed as follows: first, the image-level semantic representation and each semantic prototype are linearly mapped and normalized, and then the corresponding initial correlation score is obtained by calculating the vector dot product similarity.

[0012] Preferably, the implementation of step S5 includes: Step S51: Represent the attributes of the entity reasoning hybrid expert module. Perform weight modulation; Step S52: Characterize the attributes after weight modulation Perform weighted fusion to generate a visual representation ; Step S53: Use the visual representation Z as the input feature of the decoder, and perform sequence modeling and semantic reasoning on the visual representation Z through the decoder to generate a text description corresponding to the drainage fluid image I; Step S54: Generate the final drainage clinical report based on the generated text description.

[0013] A drainage fluid monitoring system based on an entity reasoning hybrid expert module includes: The data processing module is used to acquire images of drainage fluid and extract the spatial visual feature sequence of the drainage fluid images accordingly. The entity reasoning hybrid expert module is connected to the data processing module and processes the spatial visual feature sequence to obtain attribute representations; The structured entity prototype module initializes the structured entity prototype and obtains the semantic prototype. The hierarchical routing module is connected to the entity reasoning hybrid expert module and the structured entity prototype module, and couples spatial visual feature sequences, attribute representations and semantic prototypes to calculate fusion weights. The report generation module, connected to the hierarchical routing module, is used to perform weighted fusion of fusion weights and attribute representations to generate a visual representation, and to decode the visual representation to obtain the referral clinical report.

[0014] The present invention has the following advantages over the prior art: 1. This invention designs an entity reasoning hybrid expert module, which is specifically modeled for the unique heterogeneous clinical semantics of drainage fluid, and achieves deep alignment between clinical prior knowledge and visual features.

[0015] 2. This invention introduces a structured entity prototype and a hierarchical routing mechanism. It uses the PubMedBERT pre-trained model to transform abstract clinical definitions into semantic embedding vectors and directly injects this prior knowledge into the routing decision. It can dynamically allocate the participation of each expert according to the image content, ensuring that the generated referral clinical report is highly consistent with the clinical logic at the entity level, avoiding the logical confusion or missing key indicators common in traditional methods.

[0016] 3. The drainage clinical report generated by this invention has reached a high level in terms of content richness, clinical indicator alignment and image description accuracy. This system can not only significantly reduce the workload of medical staff in manual recording and reduce medical errors, but also realize 24-hour all-weather automated monitoring of perioperative drainage, which has important clinical early warning significance for the early detection of key and even fatal complications. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the execution of the drainage fluid monitoring method of the present invention; Figure 2 This is a schematic diagram of the drainage fluid monitoring system of the present invention. Detailed Implementation

[0018] The present invention will now be further described with reference to the accompanying drawings and specific embodiments: To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described below with reference to the accompanying drawings and embodiments, and relevant terms will be explained.

[0019] Example 1: like Figure 1 As shown, a method for monitoring drainage fluid based on an entity reasoning hybrid expert module includes the following steps: Step S1: Obtain the drainage fluid image I to be identified, and extract the spatial visual feature sequence V of the drainage fluid image I using a pre-trained backbone network. This pre-trained backbone network can use an existing ResNet-101 network; this invention preferably uses a ResNet-101 network as the feature extraction backbone, but other pre-trained networks can also be used, as long as they can extract the spatial visual feature sequence V. The specific implementation of this step is as follows: Step S11: Acquire a visible light image of the drainage bag using an image acquisition device to obtain the drainage fluid image I to be identified; wherein, the drainage fluid image I is two-dimensional image data used to characterize the appearance information of the drainage bag and the drainage fluid. Step S12: Process the drainage fluid image I into a format that meets the input requirements of the pre-trained backbone network. The processing method includes one or more operations, such as size normalization / scaling, pixel value normalization, channel format alignment, etc., to obtain the adapted drainage fluid image I'. Specifically, the drainage fluid image I is first resized to 256*256 pixels, then randomly cropped to 224*224 pixels, and data augmentation is performed by horizontal and vertical flipping. The pixel values ​​also need to be normalized using the dataset-specific mean and standard deviation. Step S13: Input the adapted drainage fluid image I' from step S12 into the pre-trained backbone network, and perform forward calculation on the drainage fluid image I' through the convolution and residual structure of the backbone network to obtain the spatial feature response map corresponding to the drainage fluid image I'. Step S14: Expand the spatial feature response map in step S13 in terms of spatial dimensions, for example, by flattening or rearranging it according to spatial location, and record the feature vector corresponding to each spatial location as a visual word, thereby forming a spatial visual feature sequence V; output the formed spatial visual feature sequence V for subsequent recognition and analysis of the drainage fluid image I.

[0020] Step S2: Construct an entity reasoning hybrid expert module. Input the spatial visual feature sequence V into the entity reasoning hybrid expert module and extract the attribute representations corresponding to the entity reasoning hybrid expert module. The entity reasoning hybrid expert module includes a global context expert module, a fluid dynamics expert module, and a local morphology expert module. The functions of each module are as follows: The global context expert module leverages Mamba's global modeling capabilities to effectively capture clues about the configuration of the drainage device and overall complications, solving the problem of long-distance spatial dependence. The fluid dynamics expert module, through an activation graph-guided mechanism, can adaptively focus on the liquid region, significantly improving the perception accuracy of the color, volume, and clarity of the drainage liquid. The local morphology expert module, through multi-scale convolutional pathways, enhances the sensitivity to small and hidden structures such as pipe wall attachments and sediments, and overcomes the limitation of a single feature extractor in simultaneously taking into account both global context and fine local features.

[0021] Among them, attribute representation in the above steps The extraction methods are as follows: By utilizing the constructed global context expert module, which is configured as a sequence modeling structure with global dependency modeling capabilities, the long-range spatial correlation of the spatial visual feature sequence V is aggregated and enhanced. During execution, the spatial visual feature sequence V is input into the global context expert module to obtain the context-enhanced attribute representation. : Formula 1 Mamba, mentioned here, is a deep learning network based on a state-space model. It excels at capturing long-range global context information. Initially proposed in 2023 by a research team from Carnegie Mellon University and Princeton University, it aimed to address the efficiency bottleneck of traditional Transformer models when processing long sequence data. Its core innovation lies in the Selective State Space Model (S3M), which significantly improves computational efficiency and long sequence modeling capabilities by dynamically adjusting information transmission paths. Alternatively, one can directly cite Mamba from the paper "Linear-Time Sequence Modeling with Selective State Spaces".

[0022] By utilizing the constructed liquid dynamics expert module, which is configured as a feature recalibration structure with region emphasis / reweighting capabilities, the representation contribution of the spatial visual feature sequence V to adaptively highlight fluid-related regions is achieved. During execution, the spatial visual feature sequence V is input into the liquid dynamics expert module, and a region response is generated through nonlinear activation. Element-wise modulation and projection transformation are then performed on the spatial visual feature sequence V to obtain the attribute representation. : Formula 2 In the formula, C represents the number of channels of V. express Visual features of the c-th channel, c Here, c represents the classifier weights, and c is a non-zero natural number. By utilizing the constructed local morphology expert module, which is configured as a multi-scale local representation extraction structure, the ability to distinguish fine-grained spatial local structures is improved. During execution, the spatial visual feature sequence V is input into the local morphology expert module, and local features under different receptive fields are obtained through multi-branch convolutional filtering. Spatial attention weights are generated for each branch for weighted fusion to obtain attribute representations. ; Formula 3 Formula 4 Formula 5 Where k represents the size of three different convolutional kernels: 3, 5, and 7. Representatives utilize Convolution of a kernel, for example This represents a convolution with a 3x3 kernel. This represents a convolution using a 3x3 kernel. This represents intermediate variables in the calculation process. Represents average pooling. Represents max pooling. This represents the MLP projection layer, i.e., the multilayer perceptron module.

[0023] It is understandable that the above steps involve parallel computation performed by three expert modules, which then output three types of attribute representations respectively. This enables forward computation of the spatial visual feature sequence V, and the three elements need to be aligned in terms of representation dimension for subsequent unified processing or fusion.

[0024] Step S3: Constructing a structured entity prototype Previously, it was necessary to pre-establish an attribute labeling system based on the clinical monitoring scenario of drainage fluid, and pre-define nine drainage clinical attributes and their corresponding entity sets; then, the PubMedBERT pre-trained model was used to semantically embed the entity sets corresponding to each drainage clinical attribute, and to create structured entity prototypes. Initialization is performed, and a semantic prototype corresponding to the hybrid expert module for entity reasoning is generated through attention aggregation. ; Step S31: Based on the report generation requirements of the clinical monitoring task of drainage fluid, a labeling system for nine clinical attributes of drainage is pre-established; the nine clinical attributes of drainage include drainage source, drainage tube type, complication type, drainage fluid volume, drainage fluid clarity, drainage fluid color, deposits on the drainage bag wall, sediments in the drainage fluid, and sediment color; Step S32: For each drainage clinical attribute, based on the clinical report writing standards, medical terminology definitions and the annotation content in the training data, organize multiple medical entity terms corresponding to the drainage clinical attribute to form a corresponding attribute entity set, which serves as the basis for the structured semantic description of the drainage clinical attribute. Step S33: Input the attribute entity sets corresponding to each clinical attribute of drainage into the PubMedBERT pre-trained model for encoding to obtain semantic embedding vectors corresponding to the medical entity terms in each attribute entity set. and form a structured entity prototype. .in, Refers to the ordinal numbers of the nine clinical attributes of drainage. Refers to the first The number of entities possessed in each clinical attribute for drainage, n being [1, N]. i ]; Step S34: Evaluate the structured entity prototype using an attention aggregation generation method. Importance weights; specifically, the structured entity prototype Each semantic embedding vector Perform linear transformations on each vector to map them to a unified semantic representation space; then, convert the linearly transformed semantic embedding vectors... The semantic query information is correlated with the corresponding clinical attributes of the target audience to obtain the original score for each medical entity term. Then, each original score is normalized within the same entity semantic set to obtain the normalized weight for each medical entity term. The normalized weight is used to characterize each medical entity term within the current structured entity prototype. The relative contribution in; Step S35: Based on the normalized weights obtained in step S34, adjust the semantic embedding vectors corresponding to each medical entity term. Weighted aggregation is performed; specifically, the semantic embedding vectors of each medical entity term after linear transformation are normalized and then summed according to the corresponding normalization weights to generate an attribute semantic indicator vector corresponding to the current clinical attribute of the traffic acquisition; the attribute semantic indicator vector is used to summarize the overall semantic features of the clinical attribute of the traffic acquisition. Step S36: Classify the drainage clinical attributes based on the differences in the attribute semantic indicator vectors. Drainage source, drainage tube type, and complication type constitute one category, corresponding to the overall context expert module in step S21; drainage fluid volume, drainage fluid clarity, and drainage fluid color constitute another category, corresponding to the fluid dynamics expert module in step S22; drainage bag wall deposits, sediments in the drainage fluid, and sediment color constitute another category, corresponding to the local morphology expert module in step S23. Step S37: Based on the classification results, establish the correspondence between the attribute semantic indicator vector and the entity reasoning hybrid expert module. Within each classification, the attribute semantic indicator vector corresponding to the clinical attributes in the current classification is weighted and summed again using normalized weights to generate the semantic prototype corresponding to the entity reasoning hybrid expert module. Different semantic groups form different semantic prototypes, which are used to represent the structured semantic priors corresponding to the hybrid expert module for entity reasoning.

[0025] Step S4: Sequence of spatial visual features Attribute representation With semantic prototype Couple the components and calculate the fusion weights corresponding to the hybrid expert modules for each entity reasoning. ; Step S41: Perform global aggregation processing on the spatial visual feature sequence V to form an image-level semantic representation that characterizes the overall semantic information of the current input image; then combine the image-level semantic representation with the three semantic prototypes respectively. Relevance calculation is performed to obtain initial relevance scores corresponding to the three semantic prototypes, which are used to characterize the degree of matching between the entity reasoning hybrid expert module and the overall semantics of the current input image; wherein, the specific method of relevance calculation is as follows: first, the image-level semantic representation and each semantic prototype are linearly mapped and normalized, and then the corresponding initial relevance scores are obtained by calculating the vector dot product similarity. Step S42: Represent the attributes respectively The aggregation process, also known as Average Pooling, is performed to obtain the overall feature representation, which is then compared with the corresponding semantic prototype. Feature coupling is performed to construct a joint feature representation that simultaneously incorporates visual attribute information and structured semantic priors. This feature coupling method concatenates the overall feature representation corresponding to the entity reasoning hybrid expert module with its corresponding semantic prototype along the feature dimension. The coupled joint feature representation is then normalized and subjected to nonlinear transformation before being input into a gating network for computation. This yields fine-grained relevance scores for each entity reasoning hybrid expert module, used to characterize the semantic consistency between the attribute representations of the entity reasoning hybrid expert module and its corresponding semantic prototype. Normalization is used to unify the scale range of the joint feature representation, and the nonlinear transformation preferably employs an activation function to map the normalized joint feature representation, thereby enhancing the gating network's ability to represent nonlinear semantic relationships. Step S43: The initial relevance score obtained in step S41 and the fine-grained relevance score obtained in step S42 are weighted and fused to obtain the fusion score corresponding to the entity reasoning hybrid expert module respectively; then the fusion score is normalized by the Softmax function to obtain the fusion weight corresponding to each entity reasoning hybrid expert module. Weighted fusion refers to the weighted summation of the initial relevance score and the fine-grained relevance score according to a preset fusion coefficient. The fusion weight represents the contribution ratio of the entity reasoning hybrid expert module in the current input sample and serves as the output of the entity-guided hierarchical routing mechanism for subsequent attribute representation. Weighted fusion processing.

[0026] Step S5: Combine the fusion weights corresponding to the expert modules for each entity reasoning. and the corresponding attribute representation By fusing the data, a visual representation can be generated. And use a decoder to decode the visual representation. Generate a clinical report on drainage.

[0027] Step S51: Attribute representation of each entity reasoning hybrid expert module Weight modulation is performed. This involves using the fusion weights separately. Attribute representation Element-wise weight modulation is performed so that the representation of each attribute is scaled proportionally according to its fusion weight, in order to highlight the expert modules with higher contributions and suppress the expert modules with lower contributions. Step S52: Characterize the attributes after weight modulation Weighted fusion is performed. The three attribute representations, after weight modulation, are weighted and summed to generate a unified visual representation. Among them, the unified visual representation Z comprehensively reflects the attribute information extracted by the three entity reasoning hybrid expert modules and is used to represent the overall visual semantic features of the input image; Step S53: Use the unified visual representation Z as the input feature of the decoder, and perform sequence modeling and semantic reasoning on the visual representation through the decoder to gradually generate text descriptions corresponding to the input images; Step S54: Generate the final drainage clinical report based on the text description output by the decoder, which describes the drainage-related clinical attribute information.

[0028] Example 2: like Figure 2 As shown, the present invention also discloses a drainage fluid monitoring system based on an entity reasoning hybrid expert module, the drainage fluid monitoring system comprising: A data processing module is used to acquire drainage fluid images and extract corresponding spatial visual feature sequences from the drainage fluid images; the data processing module further includes: The image acquisition module is used to acquire visible light images of the drainage bag to form an image I of the drainage fluid to be identified; An adaptation processing module is used to process the drainage fluid image I, converting it into a format that meets the input requirements of the pre-trained backbone network. The forward computation module is used to perform forward computation on the drainage fluid image I' to obtain the spatial feature response map; The spatial dimension unfolding module is used to unfold the spatial feature response map in spatial dimensions to form a spatial visual feature sequence V.

[0029] An entity reasoning hybrid expert module, connected to the data processing module, processes the spatial visual feature sequence to obtain attribute representations; wherein, the entity reasoning hybrid expert module further includes: The overall context expert module is used to obtain context-enhanced attribute representations. ; The liquid dynamics expert module is used to perform element-wise modulation and projection transformation on the spatial visual feature sequence V to obtain attribute representations. ; The local morphology expert module is used to generate spatial attention weights for each branch for weighted fusion to obtain attribute representations. .

[0030] The structured entity prototype module initializes the structured entity prototype and obtains the semantic prototype; the structured entity prototype module further includes: The encoding module is used to input each attribute entity set into the PubMedBERT pre-trained model for encoding, and to obtain the semantic embedding vectors corresponding to the medical entity terms in each attribute entity set, forming a structured entity prototype. The evaluation module is used to assess the importance weights of structured entity prototypes. The weighted aggregation module is used to perform weighted aggregation on the semantic embedding vectors corresponding to each medical entity term to generate an attribute semantic indicator vector corresponding to the current clinical attributes of the traffic flow. The classification module is used to classify based on differences in attribute semantic indicator vectors; The semantic prototype generation module is used to establish the correspondence between the attribute semantic indicator vector and the entity reasoning hybrid expert module based on the classification results, and generate three semantic prototypes.

[0031] A hierarchical routing module, connected to the entity reasoning hybrid expert module and the structured entity prototype module, couples spatial visual feature sequences, attribute representations, and semantic prototypes to calculate fusion weights; wherein, the hierarchical routing module further includes: The initial correlation scoring module is used to perform global aggregation processing on the spatial visual feature sequence V to form an image-level semantic representation, and to calculate the correlation between the image-level semantic representation and the semantic prototype to obtain the initial correlation score. The fine-grained relevance scoring module is used to aggregate attribute representations to obtain an overall feature representation, and then couple it with semantic prototypes to construct a joint feature representation; and then normalize and nonlinearly transform the joint feature representation to obtain a fine-grained relevance score. The weighted fusion calculation module is used to perform weighted fusion of the initial correlation score and the fine-grained correlation score to obtain the fusion score and the fusion weight.

[0032] A report generation module, connected to the hierarchical routing module, is used to perform weighted fusion of fusion weights and attribute representations to generate a visual representation, and to decode the visual representation to obtain the referral clinical report; wherein, the report generation module further includes: The weight modulation module is used to modulate the weights of the attribute representations. The visual representation generation module is used to perform weighted fusion of the weighted attribute representations to generate visual representations. The decoding module is used for sequence modeling and semantic reasoning of the visual representation to generate a text description corresponding to the drainage fluid image I.

[0033] Furthermore, the present invention also discloses a computer-readable storage medium including a memory, which facilitates the execution of the automatic drainage report generation method in a computer by storing the drainage report generation method in the memory.

Claims

1. A method for monitoring drainage fluid based on a hybrid expert module with entity reasoning, characterized in that, Includes the following steps: Step S1: Obtain the drainage fluid image I to be identified, and extract the spatial visual feature sequence V of the drainage fluid image I using a pre-trained backbone network; Step S2: Construct an entity reasoning hybrid expert module, which includes a global context expert module, a fluid dynamics expert module, and a local morphology expert module; input the spatial visual feature sequence V into the entity reasoning hybrid expert module, and extract the attribute representations corresponding to the entity reasoning hybrid expert module. ; Step S3: Construct and initialize a structured entity prototype to generate a semantic prototype corresponding to the entity reasoning hybrid expert module. ; Step S4: Sequence of spatial visual features Attribute representation With semantic prototype Couple the components and calculate the fusion weights corresponding to the hybrid expert module for entity reasoning. ; Step S5: Combine the fusion weights corresponding to the entity reasoning hybrid expert module. and attribute representation By fusing the data, a visual representation can be generated. The visual representation Z is decoded using a decoder to generate a clinical report on drainage.

2. The method for monitoring drainage fluid based on an entity reasoning hybrid expert module according to claim 1, characterized in that, The implementation of step S1 includes: Step S11: Acquire a visible light image of the drainage bag using an image acquisition device to obtain the drainage fluid image I to be identified; Step S12: Process the drainage fluid image I into a format that meets the input requirements of the pre-trained backbone network to obtain the adapted drainage fluid image I'; Step S13: Input the adapted drainage fluid image I' into the pre-trained backbone network for forward computation to obtain the spatial feature response map corresponding to the drainage fluid image I'; Step S14: Expand the spatial feature response map in terms of spatial dimensions to form a spatial visual feature sequence V; The pre-trained backbone network is a ResNet-101 network.

3. The method for monitoring drainage fluid based on an entity reasoning hybrid expert module according to claim 2, characterized in that, The processing methods in step S12 include one or more of the following: size normalization / scaling, pixel value normalization, and channel format alignment. The implementation method of spatial dimension unfolding in step S14 is as follows: flatten or rearrange according to spatial position, and record the feature vector corresponding to each spatial position as a visual word.

4. The method for monitoring drainage fluid based on an entity reasoning hybrid expert module according to claim 1, characterized in that, Attribute representation in step S2 The extraction methods are as follows: The spatial visual feature sequence V is input into the overall context expert module to obtain the context-enhanced attribute representation. : Official 1 Mamba represents a deep learning network based on a state-space model that captures long-range global contextual information. The spatial visual feature sequence V is input into the fluid dynamics expert module. A region response is generated through nonlinear activation, and the spatial visual feature sequence V is subjected to element-wise modulation and projection transformation to obtain the attribute representation. : Official 2 In the formula, C represents the number of channels of V. express Visual features of the c-th channel, c Here, c represents the classifier weights, and c is a non-zero natural number. The spatial visual feature sequence V is input into the local morphology expert module, where local features under different receptive fields are obtained through multi-branch convolutional filtering. Spatial attention weights are generated for each branch for weighted fusion to obtain the attribute representation. ; Official 3 Official 4 Official 5 In the formula, k represents the size of three different convolution kernels: 3, 5, and 7. Representatives utilize Convolution of convolution kernels; This represents intermediate variables in the calculation process. Represents average pooling. Represents max pooling. This represents the MLP projection layer, i.e., the multilayer perceptron module.

5. The method for monitoring drainage fluid based on a hybrid expert module with entity reasoning according to claim 1, characterized in that, The implementation of step S3 includes: Step S31: Based on the report generation requirements of the clinical monitoring task of drainage fluid, a labeling system for nine clinical attributes of drainage is pre-established; the nine clinical attributes of drainage include drainage source, drainage tube type, complication type, drainage fluid volume, drainage fluid clarity, drainage fluid color, deposits on the drainage bag wall, sediments in the drainage fluid, and sediment color; Step S32: For each clinical attribute of the patient's flow, based on the clinical report writing standards, medical terminology definitions, and the annotation content in the training data, organize multiple medical entity terms corresponding to the current clinical attribute of the patient's flow to form a corresponding attribute entity set; Step S33: Input the attribute entity sets corresponding to each clinical attribute of drainage into the PubMedBERT pre-trained model for encoding to obtain semantic embedding vectors corresponding to the medical entity terms in each attribute entity set. and form a structured entity prototype. ;in, Refers to the ordinal numbers of the nine clinical attributes of drainage. Refers to the first The number of entities possessed in each clinical attribute for drainage, n being [1, N]. i ]; Step S34: Evaluate the structured entity prototype using an attention aggregation generation method. Importance weights; Step S35: Based on the normalized weights obtained in step S34, adjust the semantic embedding vectors corresponding to each medical entity term. Perform weighted aggregation to generate an attribute semantic indicator vector corresponding to the current clinical attributes of the drainage. Step S36: Classify based on the differences in the semantic indicator vectors of the clinical attributes of drainage; Step S37: Based on the classification results, establish the correspondence between the attribute semantic indicator vector and the entity reasoning hybrid expert module, and generate the semantic prototype corresponding to the entity reasoning hybrid expert module. .

6. The method for monitoring drainage fluid based on an entity reasoning hybrid expert module according to claim 1, characterized in that, The implementation of step S34 includes: This structured entity prototype Each semantic embedding vector Perform linear transformations on each vector to map them to a unified semantic representation space; then, convert the linearly transformed semantic embedding vectors... The semantic query information is correlated with the corresponding clinical attributes of the target audience to obtain the original score for each medical entity term. Then, each original score is normalized within the same entity semantic set to obtain the normalized weight for each medical entity term. The normalized weight is used to characterize each medical entity term within the current structured entity prototype. The relative contribution of [the entity / entity].

7. The method for monitoring drainage fluid based on an entity reasoning hybrid expert module according to claim 1, characterized in that, The implementation of step S4 includes: Step S41: Perform global aggregation processing on the spatial visual feature sequence V to form an image-level semantic representation that represents the overall semantic information of the current input image; then combine the image-level semantic representation with the three semantic prototypes. Perform relevance calculations to obtain initial relevance scores corresponding to the three semantic prototypes; Step S42: Represent the attributes respectively The aggregation process is performed to obtain the corresponding overall feature representation, which is then compared with the corresponding semantic prototypes. Feature coupling is performed to construct a joint feature representation that simultaneously includes visual attribute information and structured semantic priors; then the joint feature representation after feature coupling is normalized and nonlinearly transformed, and input into a gating network for calculation to obtain fine-grained relevance scores corresponding to the entity reasoning hybrid expert module respectively. Step S43: The initial relevance score obtained in step S41 and the fine-grained relevance score obtained in step S42 are weighted and fused to obtain the fusion score corresponding to the entity reasoning hybrid expert module respectively; then the fusion score is normalized by the Softmax function to obtain the fusion weight corresponding to each entity reasoning hybrid expert module. .

8. The method for monitoring drainage fluid based on a hybrid expert module for entity reasoning according to claim 7, characterized in that, The correlation calculation method in step S41 is as follows: first, the image-level semantic representation and each semantic prototype are linearly mapped and normalized, and then the corresponding initial correlation score is obtained by calculating the vector dot product similarity.

9. The method for monitoring drainage fluid based on an entity reasoning hybrid expert module according to claim 1, characterized in that, The implementation of step S5 includes: Step S51: Represent the attributes of the entity reasoning hybrid expert module. Perform weight modulation; Step S52: Characterize the attributes after weight modulation Perform weighted fusion to generate a visual representation ; Step S53: Use the visual representation Z as the input feature of the decoder, and perform sequence modeling and semantic reasoning on the visual representation Z through the decoder to generate a text description corresponding to the drainage fluid image I; Step S54: Generate the final drainage clinical report based on the generated text description.

10. A drainage fluid monitoring system based on an entity reasoning hybrid expert module, characterized in that, include: The data processing module is used to acquire images of drainage fluid and extract the spatial visual feature sequence of the drainage fluid images accordingly. The entity reasoning hybrid expert module is connected to the data processing module and processes the spatial visual feature sequence to obtain attribute representations; The structured entity prototype module initializes the structured entity prototype and obtains the semantic prototype. The hierarchical routing module is connected to the entity reasoning hybrid expert module and the structured entity prototype module, and couples spatial visual feature sequences, attribute representations and semantic prototypes to calculate fusion weights. The report generation module, connected to the hierarchical routing module, is used to perform weighted fusion of fusion weights and attribute representations to generate a visual representation, and to decode the visual representation to obtain the referral clinical report.

Images