Surgical report generation method based on isomorphic graph and heterogeneous graph feature fusion and instrument action bias adjustment

Abstract

The application belongs to the cross technical field of surgical scene understanding and natural language processing, and specifically discloses a surgical report generation method based on isomorphic graph and heterogeneous graph feature fusion and instrument action bias adjustment. For a given surgical scene image, first, an isomorphic graph and a heterogeneous graph are constructed; then, isomorphic graph relationship encoding, heterogeneous graph relationship encoding and graph structure semantic fusion are respectively performed; in the report text generation process, the original prediction score of the candidate word is gradually calculated according to the generated predicted text sequence and the fused semantic representation, and the original prediction score of the action word in the candidate word is corrected by using the instrument action bias adjustment, and finally the surgical report text is gradually generated. The method can retain the overall structure information of the surgical scene while enhancing the expression ability of the local instrument, tissue and action interaction relationship, and introduce the instrument action bias adjustment in the text decoding stage, so that the generated surgical report is more consistent with the surgical scene semantics and the clinical description habit.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of surgical scene understanding and natural language processing, and in particular to a surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and the adjustment of instrument motion bias. Background Technology

[0002] Surgical report generation refers to the use of neural network models to understand surgical images, surgical video frames, or their extracted structured features, automatically identifying the relationships between instruments, tissues, and actions in the surgical scene, and converting this visual and structured information into natural language report text that conforms to medical expression habits. This technology can reduce the burden of manual recording and provide detailed reference for postoperative intervention. The various instruments and equipment and complex interactions in the surgical scene make generating accurate surgical reports quite challenging, and surgical reports also require a high degree of logical consistency between medical terminology and action relationships.

[0003] For surgical report generation, existing methods typically rely on a single type of graph structure for modeling. Methods using only isomorphic graphs usually treat instruments, tissues, and action relationships in the surgical scene as nodes of the same type, modeling connections between targets through uniform adjacency relationships. This can enhance the expression of relationships between targets to some extent, but the connecting edges between two nodes only represent connectivity and cannot distinguish the semantic differences between instrument, tissue, and action relationships, easily leading to insufficient relationship expression. Methods using only heterogeneous graphs typically treat instruments and tissues in the surgical scene as nodes of the same type, with action relationships as connecting edges between them. While this can achieve local perception to some extent, it fails to fully explore the potential of various interaction relationships in the graph, thus affecting the completeness and stability of the generated report. Furthermore, existing methods lack explicit constraints on the semantic correspondence between instruments and actions, easily leading to mismatches between action relationships and instruments in the generated text, thus affecting the accuracy of the surgical report.

[0004] Therefore, proposing a surgical report generation method that can jointly utilize isomorphic and heteromorphic graphs and introduce instrument action bias adjustment during the report generation process, in order to fully model the multi-level interactive information between instruments, tissues and action relationships in the surgical scene and improve the accuracy of the generated surgical report, has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and the adjustment of instrument motion bias, so as to solve the problems existing in the prior art.

[0006] This invention provides a surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias, comprising:

[0007] S1: Acquire surgical scene images, identify instrument entities and tissue entities in the images, and obtain the interaction actions between instrument entities and tissue entities; construct a heterogeneous graph with instrument entities and tissue entities as nodes and interaction actions as relation edges; transform the action relation edges of the heterogeneous graph into independent action nodes, and establish connected edges between instrument nodes and corresponding action nodes, and between action nodes and corresponding tissue nodes to obtain a homogeneous graph.

[0008] S2: The isomorphic graph is processed using isomorphic graph relation encoding to obtain an isomorphic graph semantic representation used to characterize the overall relationship of the target in the surgical scene;

[0009] S3: The heterogeneous graph is processed using heterogeneous graph relation encoding to obtain a heterogeneous graph semantic representation used to characterize the local interaction information between instruments, tissues and actions;

[0010] S4: The isomorphic graph semantic representation and the heteromorphic graph semantic representation are fused to obtain a fused semantic representation;

[0011] S5: Utilize the fused semantic representation and the generated predicted text sequence to predict candidate words step by step, and obtain the original prediction scores of the candidate words for the corresponding time step. After each time step prediction is completed, the original prediction scores of the action words in the candidate words of that time step are adjusted for device action bias based on the generated predicted text sequence and the learned device-action correspondence. The output words for the corresponding time step are determined based on the adjusted scores, and the generated predicted text sequence is updated until the final surgical report is generated.

[0012] Preferably, S2 specifically includes:

[0013] S21: Perform a linear transformation on the features of the isomorphic graph nodes to obtain the query matrix. Key matrix Sum matrix ;

[0014] S22: Based on the query matrix Bond matrix Calculate the attention weight matrix between nodes in the isomorphic graph. ;

[0015] S23: Based on the attention weight matrix , value matrix Weighted fusion is performed to obtain a node representation that incorporates information from other relevant nodes. ;

[0016] S24: Represent the nodes Further processing via residual connections, normalization, and feedforward neural networks yields an isomorphic graph semantic representation for characterizing the overall relationships between targets in a surgical scenario.

[0017] Further optimization, S3 specifically includes:

[0018] S31: For each node in the heterogeneous graph, calculate the aggregation relationship information of that node based on its neighboring nodes and the corresponding action relationship edges. ;

[0019] S32: Move the node The features and aggregation relationship information are gated and fused to obtain the features with enhanced relationships.

[0020] S33: Input the enhanced features into the feedforward neural network for further feature transformation, and perform residual connection and normalization processing on the output of the feedforward neural network and the enhanced features to obtain the updated node features. All the updated node features together form the heterogeneous graph semantic representation.

[0021] Further optimization, S4 specifically includes:

[0022] S41: Using isomorphic graph semantic representation as query input and heteromorphic graph semantic representation as key and value input, respectively, linear transformations are performed to obtain the fused query matrix. Fusion bond matrix and fusion value matrix ;

[0023] S42: Based on the fusion query matrix and fusion bond matrix Calculate the fusion attention weight matrix between isomorphic and heteromorphic graph semantic representations. ;

[0024] S43: Based on the fusion attention weight matrix For the fusion value matrix Perform weighted fusion to obtain fused information. ;

[0025] S44: Information fusion Normalization is performed to obtain the fused semantic representation. .

[0026] Further optimization, in S5, the fused semantic representation and the generated predicted text sequence are used to perform the first... The steps for predicting candidate words at each time step are as follows:

[0027] S501: with Using this as the starting marker, and combining it with the candidate words generated in the previous time step, the 1st time step is constructed. The generated predicted text sequence corresponding to each time step :

[0028] ;

[0029] in, This represents the candidate words generated at the first time step. Indicates the first Candidate words generated in -1 time step;

[0030] S502: For text sequences Word embedding and position embedding are performed to obtain the text input representation. :

[0031] ;

[0032] in, This indicates word embedding operations. This indicates a position embedding operation;

[0033] S503: Text input representation Perform self-attention calculation to obtain the text context representation. ;

[0034] S504: Represented in textual context As query information, incorporating semantic representation Using these as keys and values, cross-attention calculation is performed to obtain the candidate word prediction state. ;

[0035] S505: Predict the status of candidate words After processing through a feedforward neural network, residual connections, and normalization, the final predicted state is obtained. ;

[0036] S506: Based on the final predicted state Calculate the raw prediction score for each candidate word. :

[0037] ;

[0038] in, These are learnable parameters.

[0039] Further optimization, in S5, the predicted text sequence and the learned correspondence between the device and the action are used to refine the... The steps for adjusting the device action bias based on the original prediction scores of action words in the candidate words at each time step are as follows:

[0040] S511: Based on the text sequence Identify the device information that has already appeared and obtain the device identifier. :

[0041] ;

[0042] in, This indicates a device identification operation used to identify devices from a text sequence. Identify the instruments that have already appeared;

[0043] S512: Obtain the learned machine motion bias matrix ,in, For learnable parameter matrix, elements in Indicating instruments With action verbs The degree of matching between them is scored; the higher the degree of matching, the better. The higher the score, the better;

[0044] S513: Adjust the original predicted scores of action words in candidate words for device-action bias by using the matching degree scores between devices and action words in the device-action bias matrix:

[0045] ;

[0046] in, Indicates the first Action words for each time step The original predicted score, This represents the bias-adjusted prediction score for the action verb. This represents the bias strength parameter, which keeps the prediction score unchanged for non-action word candidate words.

[0047] Further optimization, in S5, the number of... The process of generating the final surgical report by updating the generated predicted text sequence at each time step, including the following steps, involves the following steps:

[0048] S521: Based on the candidate word prediction scores adjusted for machine motion bias, obtain the first... Set of candidate word prediction probabilities at each time step :

[0049] ;

[0050] in, Indicates the first The set of candidate word prediction scores after action word bias adjustment at each time step. The prediction scores of action word candidate words will be adjusted, while the prediction scores of non-action word candidate words will remain unchanged.

[0051] S522: Predict the probability set based on candidate words Select the candidate word with the highest predicted probability as the first... Output words at each time step :

[0052] ;

[0053] in, This means selecting the candidate word with the highest probability from the set of candidate word probabilities as the output word;

[0054] S523: The first Output words at each time step Add to text sequence The updated text sequence is obtained. :

[0055] ;

[0056] S524: Determine the generated output words Is it an end marker or an updated text sequence? If the preset maximum text length has been reached, the updated text sequence is reorganized, and the start and end markers are removed to obtain the final surgical report text; otherwise, candidate word prediction is performed for the next time step.

[0057] The surgical report generation method provided by this invention, based on the fusion of isomorphic and heteromorphic graph features and instrument action bias adjustment, first obtains the corresponding heteromorphic and isomorphic graphs for a given input surgical scene image; then, it extracts overall relationship information and local instrument, tissue, and action interaction relationship information by performing isomorphic graph relationship encoding and heteromorphic graph relationship encoding respectively; further, it performs graph structure semantic fusion to obtain a fused semantic representation, which is used as the semantic representation for text decoding; during the report text generation process, the original prediction scores of candidate words are calculated step by step based on the generated predicted text sequence and the fused semantic representation, while the original prediction scores of action words in the candidate words are corrected by instrument action bias adjustment, and finally the surgical report text is generated step by step.

[0058] The surgical report generation method provided by this invention, based on the fusion of isomorphic and heteromorphic graph features and the adjustment of instrument action bias, can enhance the ability to express the interaction relationship between local instruments, tissues and actions while retaining the overall structural information of the surgical scene. Furthermore, the introduction of instrument action bias adjustment in the text decoding stage makes the generated surgical report more in line with the semantics of the surgical scene and clinical description habits. Attached Figure Description

[0059] Figure 1 The overall flowchart of the surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and the adjustment of instrument motion bias provided by the present invention is shown below.

[0060] Figure 2 The quantitative comparison results of the method provided in this invention with methods such as Transformer, CIDA, GET, LSTNet, and SGT in the surgical report generation task. Detailed Implementation

[0061] The present invention will be further described below with reference to specific embodiments.

[0062] like Figure 1 As shown, this invention provides a surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias, including the following steps:

[0063] S1: Acquire surgical scene images, identify instrument entities and tissue entities in the images, and obtain the interaction actions between instrument entities and tissue entities; construct a heterogeneous graph with instrument entities and tissue entities as nodes and interaction actions as relation edges; transform the action relation edges of the heterogeneous graph into independent action nodes, and establish connected edges between instrument nodes and corresponding action nodes, and between action nodes and corresponding tissue nodes to obtain a homogeneous graph.

[0064] Among them, heterogeneous graphs Specifically as follows:

[0065] ;

[0066] in, It is a set of instrument nodes and tissue nodes. This is the set of action relationship edges between instrument nodes and tissue nodes. For the nodal characteristics of instruments and tissues, Edge features representing action relationships;

[0067] Preferably, the node features of the instrument and tissue are obtained by extracting features from the instrument region and tissue region respectively by the ResNet18 network, and the edge features of the action relationship are obtained by extracting features from the image region that simultaneously contains the corresponding instrument and tissue by the ResNet18 network.

[0068] Wherein, the isomorphic graph Specifically as follows:

[0069] ;

[0070] in, For a set of nodes in an isomorphic graph, , This represents the set of instrument nodes and tissue nodes in a heterogeneous graph. For the set of action relationship edges in a heterogeneous graph The converted set of action nodes; Let be the set of all connected edges in the isomorphic graph; The node features of the isomorphic graph are composed of instrument, tissue, and action relationship features. ;

[0071] S2: The isomorphic graph is processed using isomorphic graph relation encoding to obtain an isomorphic graph semantic representation used to characterize the overall relationship of the target in the surgical scene;

[0072] S2 specifically includes:

[0073] S21: Features of isomorphic graph nodes Perform a linear transformation to obtain the query matrix. Key matrix Sum matrix :

[0074] ;

[0075] ;

[0076] ;

[0077] in, , , These are learnable parameters;

[0078] S22: Based on the query matrix Bond matrix Calculate the attention weight matrix between nodes in the isomorphic graph. :

[0079] ;

[0080] in, This represents the probability normalization function;

[0081] S23: Based on the attention weight matrix , value matrix Weighted fusion is performed to obtain a node representation that incorporates information from other relevant nodes. :

[0082] ;

[0083] S24: Represent the nodes Further processing via residual connections, normalization, and feedforward neural networks yields an isomorphic graph semantic representation for characterizing the overall relationships between targets in a surgical scene. ;

[0084] S3: The heterogeneous graph is processed using heterogeneous graph relation encoding to obtain a heterogeneous graph semantic representation used to characterize the local interaction information between instruments, tissues and actions;

[0085] The specific process of S3 is as follows:

[0086] S31: For each node in the heterogeneous graph According to its neighboring nodes Calculate the aggregation relationship information of the node based on the corresponding action relationship edge. :

[0087] ;

[0088] in, Represents nodes The set of adjacent nodes, Represents nodes Number of adjacent nodes Represents a node Features Indicates adjacent nodes Features Represents a node and nodes Features of the action relationship between them This represents the hyperbolic tangent activation function. and For learnable parameters, Indicates feature splicing;

[0089] S32: Move the node Features Information related to aggregation Gated fusion is performed to obtain features with enhanced relationships. :

[0090] ;

[0091] ;

[0092] in, The gating factor is used to control the aggregation relationship information. Integrating node features To what extent, express function, This indicates a normalization operation. This indicates element-wise multiplication. and These are learnable parameters;

[0093] S33: Features that enhance the relationship The input is fed forward neural network for further feature transformation, and the output of the feedforward neural network is compared with... Residual connections and normalization are performed to obtain updated node features, which together form the heterogeneous graph semantic representation. ;

[0094] S4: The isomorphic graph semantic representation and the heteromorphic graph semantic representation are fused to obtain a fused semantic representation;

[0095] S41: Represented using isomorphic graph semantics As query input, represented by heterogeneous graph semantics The inputs are used as key and value inputs, and linear transformations are performed on them respectively to obtain the fused query matrix. Fusion bond matrix and fusion value matrix :

[0096] ;

[0097] ;

[0098] ;

[0099] in, , , These are learnable parameters;

[0100] S42: Based on the fusion query matrix and fusion bond matrix Calculate the fusion attention weight matrix between isomorphic and heteromorphic graph semantic representations. :

[0101] ;

[0102] S43: Based on the fusion attention weight matrix For the fusion value matrix Perform weighted fusion to obtain fused information. :

[0103] ;

[0104] S44: Information fusion Normalization is performed to obtain the fused semantic representation. :

[0105] ;

[0106] S5: Utilize the fused semantic representation and the generated predicted text sequence to predict candidate words step by step, and obtain the original prediction scores of candidate words for the corresponding time step. After each time step prediction is completed, the original prediction scores of action words in the candidate words of that time step are adjusted for device action bias based on the generated predicted text sequence and the learned device-action correspondence. The output words for the corresponding time step are determined based on the adjusted scores, and the generated predicted text sequence is updated until the final surgical report is generated.

[0107] Among them, the first step is to use the fused semantic representation and the generated predicted text sequence for the... The steps for predicting candidate words at each time step are as follows:

[0108] S501: with Using this as the starting marker, and combining it with the candidate words generated in the previous time step, the 1st time step is constructed. The generated predicted text sequence corresponding to each time step :

[0109] ;

[0110] in, This represents the candidate words generated at the first time step. Indicates the first Candidate words generated in -1 time step;

[0111] S502: For text sequences Word embedding and position embedding are performed to obtain the text input representation. :

[0112] ;

[0113] in, This indicates word embedding operations. This indicates a position embedding operation;

[0114] S503: Text input representation Perform self-attention calculation to obtain the text context representation. ;

[0115] S504: Represented in textual context As query information, incorporating semantic representation Using these as keys and values, cross-attention calculation is performed to obtain the candidate word prediction state. ;

[0116] S505: Predict the status of candidate words After processing through a feedforward neural network, residual connections, and normalization, the final predicted state is obtained. ;

[0117] S506: Based on the final predicted state Calculate the raw prediction score for each candidate word. :

[0118] ;

[0119] in, These are learnable parameters;

[0120] Among them, the first prediction text sequence is based on the generated prediction text sequence and the learned correspondence between the device and the action. The steps for adjusting the device action bias based on the original prediction scores of action words in the candidate words at each time step are as follows:

[0121] S511: Based on the text sequence Identify the device information that has already appeared and obtain the device identifier. :

[0122] ;

[0123] in, This indicates a device identification operation used to identify devices from a text sequence. Identify the instruments that have already appeared;

[0124] S512: Obtain the learned machine motion bias matrix ,in, For learnable parameter matrix, elements in Indicating instruments With action verbs The degree of matching between them is scored; the higher the degree of matching, the better. The higher the score, the better;

[0125] S513: Adjust the original predicted scores of action words in candidate words for device-action bias by using the matching degree scores between devices and action words in the device-action bias matrix:

[0126] ;

[0127] in, Indicates the first Action words for each time step The original predicted score, This represents the bias-adjusted prediction score for the action verb. This represents the bias strength parameter, whose prediction score remains unchanged for non-action word candidate words;

[0128] Among them, the number is determined according to the adjusted score. The process of generating the final surgical report by updating the generated predicted text sequence at each time step, including the following steps, involves the following steps:

[0129] S521: Based on the candidate word prediction scores adjusted for machine motion bias, obtain the first... Set of candidate word prediction probabilities at each time step :

[0130] ;

[0131] in, Indicates the first The set of candidate word prediction scores after action word bias adjustment at each time step. The prediction scores of action word candidate words will be adjusted, while the prediction scores of non-action word candidate words will remain unchanged.

[0132] S522: Predict the probability set based on candidate words Select the candidate word with the highest predicted probability as the first... Output words at each time step :

[0133] ;

[0134] in, This means selecting the candidate word with the highest probability from the set of candidate word probabilities as the output word;

[0135] S523: The first Output words at each time step Add to text sequence The updated text sequence is obtained. :

[0136] ;

[0137] S524: Determine the generated output words Is it an end marker or an updated text sequence? If the preset maximum text length has been reached, the updated text sequence is reorganized, and the start and end markers are removed to obtain the final surgical report text; otherwise, candidate word prediction is performed for the next time step.

[0138] The surgical report generation method provided by this invention introduces isomorphic graph relationship modeling, heteromorphic graph relationship modeling, and instrument action bias adjustment, realizing a complete technical path from surgical scene graph structure extraction, graph relationship semantic fusion to report text generation.

[0139] In the specific implementation process, given an input surgical scene image, the corresponding heterogeneous graph and isogeneous graph are first obtained; then, isogeneous graph relation encoding and heterogeneous graph relation encoding are performed respectively to extract overall relation information and local instrument, tissue and action interaction relation information; further, graph structure semantic fusion is performed to obtain fused semantic representation, which is used as the semantic representation for text decoding; in the process of generating report text, the original prediction scores of candidate words are calculated step by step according to the generated predicted text sequence and fused semantic representation, and the original prediction scores of action words in candidate words are corrected by instrument action bias adjustment, and finally the surgical report text is generated step by step.

[0140] Compared to surgical report generation methods that utilize only a single graph structure, this invention simultaneously leverages both isomorphic and heteromorphic graph structures. Isomorphic graphs characterize the overall relationships between instruments, tissues, and actions within a surgical scenario, while heteromorphic graphs preserve the local interaction semantics between instrument nodes, tissue nodes, and action relationship edges. By fusing these two approaches, the model can more comprehensively understand the target relationships and action semantics within the surgical scenario.

[0141] Compared with methods that directly generate text based solely on decoding probabilities, this invention further introduces instrument action bias adjustment during the text generation process. Based on instrument information in the text sequence and the learned correspondence between instruments and actions, the original prediction scores of candidate action words are adjusted, thereby reducing the mismatch between instrument names and action words and improving the professionalism and accuracy of the generated surgical reports.

[0142] Through the complete technical implementation path described above, this invention can enhance the ability to express the interaction relationship between local instruments, tissues and actions while retaining the overall structural information of the surgical scene. Furthermore, it introduces instrument action bias adjustment during the text decoding stage, making the generated surgical report more consistent with the semantics of the surgical scene and clinical description habits.

[0143] Figure 2 This paper presents quantitative comparative results between the method of this invention and Transformer, CIDA, GET, LSTNet, and SGT methods in the surgical report generation task. Figure 2 As can be seen, the method of the present invention outperforms the comparative methods in BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE, and CIDEr metrics. This indicates that the fusion of isomorphic and heteromorphic graph features can enhance the expressive ability of the overall relationship and the local instrument, tissue, and action relationship in the surgical scene. The adjustment of instrument action bias can improve the consistency of instrument and action descriptions in the generated text, thereby making the generated surgical report more effective in terms of semantic integrity, text accuracy, and clinical description standardization.

[0144] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The apparatus of the present invention can employ a processor of a general-purpose computer, a dedicated medical testing computer, an embedded processor, or other programmable data processing device to execute instructions to implement the functional means specified in the various sub-steps of the present invention.

[0145] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias, characterized in that, include: S1: Acquire surgical scene images, identify instrument entities and tissue entities in the images, and obtain the interaction actions between instrument entities and tissue entities; A heterogeneous graph is constructed using instrument entities and tissue entities as nodes and interactive actions as relation edges; Transform the action relationship edges of the heterogeneous graph into independent action nodes, and establish connected edges between instrument nodes and corresponding action nodes, and between action nodes and corresponding tissue nodes to obtain a homogeneous graph. S2: The isomorphic graph is processed using isomorphic graph relation encoding to obtain an isomorphic graph semantic representation used to characterize the overall relationship of the target in the surgical scene; S3: The heterogeneous graph is processed using heterogeneous graph relation encoding to obtain a heterogeneous graph semantic representation used to characterize the local interaction information between instruments, tissues and actions; S4: The isomorphic graph semantic representation and the heteromorphic graph semantic representation are fused to obtain a fused semantic representation; S5: Utilize the fused semantic representation and the generated predicted text sequence to predict candidate words step by step, and obtain the original prediction scores of the candidate words for the corresponding time step. After each time step prediction is completed, the original prediction scores of the action words in the candidate words of that time step are adjusted for device action bias based on the generated predicted text sequence and the learned device-action correspondence. The output words for the corresponding time step are determined based on the adjusted scores, and the generated predicted text sequence is updated until the final surgical report is generated.

2. The surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias according to claim 1, characterized in that, S2 specifically includes: S21: Perform a linear transformation on the features of the isomorphic graph nodes to obtain the query matrix. Key matrix Sum matrix ; S22: Based on the query matrix Bond matrix Calculate the attention weight matrix between nodes in the isomorphic graph. ; S23: Based on the attention weight matrix , value matrix Weighted fusion is performed to obtain a node representation that incorporates information from other relevant nodes. ; S24: Represent the nodes Further processing via residual connections, normalization, and feedforward neural networks yields an isomorphic graph semantic representation for characterizing the overall relationships between targets in a surgical scenario.

3. The surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias according to claim 1, characterized in that, S3 specifically includes: S31: For each node in the heterogeneous graph, calculate the aggregation relationship information of that node based on its neighboring nodes and the corresponding action relationship edges. ; S32: Move the node The features and aggregation relationship information are gated and fused to obtain the features with enhanced relationships. S33: Input the enhanced features into the feedforward neural network for further feature transformation, and perform residual connection and normalization processing on the output of the feedforward neural network and the enhanced features to obtain the updated node features. All the updated node features together form the heterogeneous graph semantic representation.

4. The surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias according to claim 1, characterized in that, S4 specifically includes: S41: Using isomorphic graph semantic representation as query input and heteromorphic graph semantic representation as key and value input, respectively, linear transformations are performed to obtain the fused query matrix. Fusion bond matrix and fusion value matrix ; S42: Based on the fusion query matrix and fusion bond matrix Calculate the fusion attention weight matrix between isomorphic and heteromorphic graph semantic representations. ; S43: Based on the fusion attention weight matrix For the fusion value matrix Perform weighted fusion to obtain fused information. ; S44: Information fusion Normalization is performed to obtain the fused semantic representation. .

5. The surgical report generation method based on isomorphic and heteromorphic graph feature fusion and instrument motion bias adjustment according to claim 1, characterized in that, In S5, the fused semantic representation and the generated predicted text sequence are used to perform the first... The steps for predicting candidate words at each time step are as follows: S501: with Using this as the starting marker, and combining it with the candidate words generated in the previous time step, the 1st time step is constructed. The generated predicted text sequence corresponding to each time step : ； in, This represents the candidate words generated at the first time step. Indicates the first Candidate words generated in -1 time step; S502: For text sequences Word embedding and position embedding are performed to obtain the text input representation. : ； in, This indicates word embedding operations. This indicates a position embedding operation; S503: Text input representation Perform self-attention calculation to obtain the text context representation. ; S504: Represented in textual context As query information, incorporating semantic representation Using these as keys and values, cross-attention calculation is performed to obtain the candidate word prediction state. ; S505: Predict the status of candidate words After processing through a feedforward neural network, residual connections, and normalization, the final predicted state is obtained. ; S506: Based on the final predicted state Calculate the raw prediction score for each candidate word. : ； in, These are learnable parameters.

6. The surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias according to claim 5, characterized in that, In S5, the first part is processed based on the generated predicted text sequence and the learned correspondence between the device and the action. The steps for adjusting the device action bias based on the original prediction scores of action words in the candidate words at each time step are as follows: S511: Based on the text sequence Identify the device information that has already appeared and obtain the device identifier. : ； in, This indicates a device identification operation used to identify devices from a text sequence. Identify the instruments that have already appeared; S512: Obtain the learned machine motion bias matrix ,in, For learnable parameter matrix, elements in Indicating instruments With action verbs The degree of matching between them is scored; the higher the degree of matching, the better. The higher the score, the better; S513: Adjust the original predicted scores of action words in candidate words for device-action bias by using the matching degree scores between devices and action words in the device-action bias matrix: ； in, Indicates the first Action words for each time step The original predicted score, This represents the bias-adjusted prediction score for the action verb. This represents the bias strength parameter, which keeps the prediction score unchanged for non-action word candidate words.

7. The surgical report generation method based on the fusion of isomorphic and heteromorphic graph features and adjustment of instrument motion bias according to claim 5, characterized in that, In S5, the number is determined based on the adjusted score. The process of generating the final surgical report by updating the generated predicted text sequence at each time step, including the following steps, involves the following steps: S521: Based on the candidate word prediction scores adjusted for machine motion bias, obtain the first... Set of candidate word prediction probabilities at each time step : ； in, Indicates the first The set of candidate word prediction scores after action word bias adjustment at each time step. The prediction scores of action word candidate words will be adjusted, while the prediction scores of non-action word candidate words will remain unchanged. S522: Predict the probability set based on candidate words Select the candidate word with the highest predicted probability as the first... Output words at each time step : ； in, This means selecting the candidate word with the highest probability from the set of candidate word probabilities as the output word; S523: The first Output words at each time step Add to text sequence The updated text sequence is obtained. : ； S524: Determine the generated output words Is it an end marker or an updated text sequence? If the preset maximum text length has been reached, the updated text sequence is reorganized, and the start and end markers are removed to obtain the final surgical report text; otherwise, candidate word prediction is performed for the next time step.

Images