Multifunction bronchoscope simulation training teaching method

Abstract

The application discloses a multifunctional bronchoscope simulation training teaching method, and belongs to the technical field of training teaching, and specifically comprises the following steps: collecting instrument moving track data and real-time video stream data in bronchoscope operation, comparing the track data with a standard path to identify a path deviation section, extracting bronchial bifurcation position marks and mucosa morphological feature marks from the video stream, establishing a space-time correlation between path deviation and anatomical features and constructing an operation logic analysis report, converting the report into semantic features and image features, inputting the semantic features and the image features into a multimodal large model for feature fusion to generate an operation rationality evaluation vector, finally analyzing the vector to obtain a path compliance score, a clinical decision score and a strain capacity score, and finally obtaining a final score result by fusing the three scores; the application effectively distinguishes between operation mistakes and reasonable clinical strain, and realizes real evaluation of the clinical decision-making ability of students.

Description

Technical Field

[0001] This invention relates to the field of training and teaching technology, specifically to a multifunctional bronchoscopy simulation training and teaching method. Background Technology

[0002] Bronchoscopy is a key interventional technique in the diagnosis and treatment of respiratory diseases, and mastering its operation skills is crucial for clinicians. Traditional skills training mainly relies on practical experience with real patients, but this method has problems such as a long learning curve, high patient safety risks, and limited teaching resources. With the development of simulation-based medical education, bronchoscopy simulation training systems have become an important teaching tool, providing trainees with a repeatable and risk-free practice environment.

[0003] Currently, existing bronchoscopy simulation training systems generally use pre-set standardized operating procedures and steps as the basis for scoring. The system tracks the movement trajectory of the instrument and compares it to a built-in ideal path to quantitatively assess the trainee's operational compliance. Such systems typically classify deviations from the pre-set path as operational errors and deduct points accordingly.

[0004] However, in actual clinical practice, physicians often need to proactively adjust their surgical approach and take necessary exploratory or avoidance actions based on individualized anatomical variations in patients or unexpected intraoperative situations (such as abnormal secretions, mucosal bleeding, or unexpected stenosis). These rationally sound decisions based on clinical judgment are often simply categorized as operational errors within the standardized scoring framework of existing training systems because they do not conform to the pre-set path. Current technology lacks the ability to deeply analyze the clinical logic and rationality behind the operations, and cannot effectively distinguish between "technical errors" and "reasonable clinical responses," resulting in scoring results that fail to truly and comprehensively reflect trainees' comprehensive decision-making abilities and adaptability in complex clinical situations. Summary of the Invention

[0005] The purpose of this invention is to provide a multifunctional bronchoscopy simulation training method to solve the following technical problems:

[0006] Current bronchoscopy simulation training systems generally use pre-set standardized pathways for scoring, classifying any deviation from the pathway as an operational error. However, in actual clinical practice, physicians often need to take necessary non-standard pathway actions based on individual patient anatomical variations or unexpected situations; these are reasonable clinical decisions. Because existing systems cannot distinguish between operational errors and reasonable clinical responses, their scoring results fail to accurately reflect the trainee's clinical decision-making abilities.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A multifunctional bronchoscopy simulation training method includes the following steps:

[0009] S1. Collect instrument movement trajectory data during bronchoscopy operation, as well as real-time video stream data of the bronchial wall.

[0010] S2. Compare the instrument movement trajectory data with the standard operating path point by point, and identify the trajectory segments that continuously deviate from the standard path as path deviation segments.

[0011] S3. Extract bronchial bifurcation location identifiers and mucosal morphological feature identifiers from real-time video stream data;

[0012] S4. Spatiotemporally correlate the path deviation sections with the corresponding bronchial bifurcation location markers and mucosal morphological feature markers to construct an operational logic analysis report;

[0013] S5. Convert the operation logic analysis report into semantic features and image features respectively, input them into the multimodal large model, and generate an operation rationality evaluation vector through feature fusion;

[0014] S6. The rationality assessment vector of the operation is analyzed to obtain the path compliance score, clinical decision score and adaptability score. The three scores are then combined to obtain the final score result.

[0015] As a further aspect of the present invention: in step S2, the specific process of identifying trajectory segments that continuously deviate from the standard path is as follows:

[0016] A sliding window averaging algorithm is applied to the instrument movement trajectory data, with the window size set to a fixed value, to calculate the spatial distance between the processed trajectory points and the corresponding points of the standard operating path.

[0017] Set a distance threshold for the exploration phase and a distance threshold for the positioning phase, wherein the distance threshold for the exploration phase is greater than the distance threshold for the positioning phase;

[0018] Determine the spatial distance sequence of continuous trajectory points. When the spatial distance between multiple consecutive points exceeds the distance threshold of the exploration phase, it is marked as an exploration deviation segment. When the spatial distance between multiple consecutive points exceeds the distance threshold of the positioning phase, it is marked as a positioning deviation segment. Merge the exploration deviation segment and the positioning deviation segment into a path deviation segment.

[0019] As a further aspect of the present invention: in step S3, the specific process of extracting the bronchial bifurcation location markers and mucosal morphological feature markers is as follows:

[0020] Gaussian pyramid decomposition is performed on video frames to generate images at multiple scales. Local extrema are detected in each scale image and used as candidate bifurcation points. The area and contour curvature of the region where the candidate bifurcation points are located are calculated. Candidate points are filtered according to area thresholds and curvature thresholds. Candidate points that meet the threshold requirements are marked as bronchial bifurcation location markers.

[0021] The mucosal region is segmented by performing a region growing algorithm on the video frame. The set features of the mucosal region are calculated, including gradient direction histogram features, local binary pattern features, and mean and variance of color channels. The set features are merged into feature vectors and normalized to generate mucosal morphology feature identifiers.

[0022] As a further aspect of the present invention: in step S4, the specific process of constructing the operation logic analysis report is as follows:

[0023] Based on the temporal distribution characteristics of the path deviation section, the operation process is divided into multiple time windows. Within each time window, the spatial distribution density of the bronchial bifurcation location markers is statistically analyzed, and the change in the feature vector of the mucosal morphological feature markers between adjacent time windows is calculated.

[0024] A spatiotemporal correlation map between the operation trajectory and the anatomical structure is established using spatial distribution density and eigenvector changes. An operation logic network is generated based on the spatiotemporal correlation map. The operation logic network contains a set of nodes and a set of edges. Nodes represent operation states, and edges represent state transitions. The operation logic network is then converted into a structured operation logic analysis report.

[0025] As a further aspect of the present invention: the generation process of the operation logic network is as follows:

[0026] The spatial distribution density and feature vector change within each time window are processed sequentially, and the spatial distribution density and feature vector change are combined into node feature vectors.

[0027] Calculate the cosine similarity between the feature vectors of adjacent time window nodes, determine the transition probability between nodes based on the cosine similarity value, and assign the transition probability value as the weight coefficient of the connecting edge;

[0028] Arrange all node feature vectors into a feature matrix, arrange all edge weights into an adjacency matrix, and combine the feature matrix and adjacency matrix to generate the operational logic network.

[0029] As a further aspect of the present invention: in step S5, the specific process of generating the operation rationality evaluation vector through feature fusion is as follows:

[0030] The operation logic network in the operation logic analysis report is time-stamp aligned and spatial coordinate matched with the real-time video stream data; the temporal change patterns of node feature vectors are extracted from the aligned operation logic network; and the spatial correspondence between bronchial image features and operation status nodes is calculated based on the spatial coordinate matching results.

[0031] The temporal variation pattern and spatial correspondence are input into the gated recurrent unit for feature fusion; the output of the gated recurrent unit is input into the multi-granularity feature interaction layer, which simultaneously processes local operation fragment features and global operation sequence features; the output of the multi-granularity feature interaction layer is input into the graph neural network for feature aggregation, and the operation rationality evaluation vector is generated based on the output of the graph neural network.

[0032] As a further aspect of the present invention: the processing procedure of the multi-granularity feature interaction layer is as follows:

[0033] Local operation segment features are decomposed into instrument motion features and anatomical environment features, and global operation sequence features are decomposed into operation trajectory features and clinical scene features;

[0034] Calculate the first correlation weight between instrument motion characteristics and clinical scenario characteristics, and calculate the second correlation weight between anatomical environment characteristics and operation trajectory characteristics;

[0035] The fusion ratio between local operation fragment features and global operation sequence features is determined based on the first and second correlation weights; the weighted fusion outputs a multi-granularity fusion feature vector.

[0036] As a further aspect of the present invention: the specific process of generating the operation rationality evaluation vector based on the output of the graph neural network is as follows:

[0037] Read the node feature vectors and edge weight matrices of the graph neural network, perform temporal pooling on the node feature vectors to extract temporal feature representations, and perform spatial pooling on the edge weight matrices to extract spatial relation representations;

[0038] The feature representations of the time and space dimensions are concatenated, and the concatenated feature representations are input into a fully connected layer for dimensionality reduction. The dimensionality-reduced features are mapped to a probability distribution through the softmax function, and the probability distribution vector is output as the operation rationality evaluation vector.

[0039] As a further aspect of the present invention: in step S6, the specific process by which the rationality assessment vector of the analytical operation is used to obtain the path compliance score, clinical decision-making score, and adaptability score is as follows:

[0040] Extract the three-dimensional trajectory coordinate sequence from the operation rationality assessment vector, and calculate the trajectory smoothness index and path efficiency coefficient; analyze the scene understanding features in the operation rationality assessment vector to generate the anatomical structure recognition accuracy and clinical protocol matching degree; analyze the temporal response features in the operation rationality assessment vector to calculate the abnormal situation detection delay and operation correction timeliness.

[0041] A path compliance score is generated based on the trajectory smoothness index and path efficiency coefficient; a clinical decision score is generated based on the accuracy of anatomical structure recognition and the clinical protocol matching degree; and an responsiveness score is generated based on the abnormal situation detection delay and the timeliness of operation correction.

[0042] As a further aspect of the present invention: the specific process for obtaining the final scoring result in step S6 is as follows:

[0043] Weighting coefficients are set according to the training stage. In the primary training stage, the weighting order is path compliance score, clinical decision score, and adaptability score. In the advanced training stage, the weighting order is adaptability score, clinical decision score, and path compliance score.

[0044] The path adherence score, clinical decision-making score, and adaptability score were normalized. The three scores were then weighted and summed according to the weighting coefficients of the corresponding training stages. The weighted sum was then mapped to the score interval to generate the final score.

[0045] The beneficial effects of this invention are:

[0046] This invention deeply integrates instrument movement trajectories with anatomical features in real-time video streams by constructing an operational logic analysis report. This establishes a spatiotemporal correlation map of path deviation sections, bronchial bifurcation locations, and mucosal morphological features, forming an operational logic network containing operational state nodes and state transition edges. Based on a multimodal large-scale model semantic and image feature fusion mechanism, combined with a multi-level processing architecture, an evaluation vector comprehensively reflecting operational rationality is generated. This vector is analyzed to obtain three independent scores: path adherence, clinical decision-making, and adaptability. A dynamic weight allocation strategy based on training stages is used for fusion evaluation. This intelligent evaluation system based on clinical scenario understanding can accurately distinguish between path deviations caused by technical unfamiliarity and reasonable operational adjustments based on clinical judgment, solving the core problem of the disconnect between mechanical scoring in traditional training systems and clinical practice. This system not only retains the evaluation function of basic operational standardization but also achieves a realistic assessment of comprehensive clinical capabilities by analyzing the operator's decision-making logic and adaptability. It provides targeted teaching feedback for trainees at different training stages, significantly improving the clinical relevance and teaching effectiveness of simulation training. Attached Figure Description

[0047] The invention will now be further described with reference to the accompanying drawings.

[0048] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Please see Figure 1 As shown, this invention is a multifunctional bronchoscopy simulation training method, comprising the following steps:

[0051] S1. Collect instrument movement trajectory data during bronchoscopy operation, as well as real-time video stream data of the bronchial wall.

[0052] S2. Compare the instrument movement trajectory data with the standard operating path point by point, and identify the trajectory segments that continuously deviate from the standard path as path deviation segments.

[0053] S3. Extract bronchial bifurcation location identifiers and mucosal morphological feature identifiers from real-time video stream data;

[0054] S4. Spatiotemporally correlate the path deviation sections with the corresponding bronchial bifurcation location markers and mucosal morphological feature markers to construct an operational logic analysis report;

[0055] S5. Convert the operation logic analysis report into semantic features and image features respectively, input them into the multimodal large model, and generate an operation rationality evaluation vector through feature fusion;

[0056] S6. The rationality assessment vector of the operation is analyzed to obtain the path compliance score, clinical decision score and adaptability score. The three scores are then combined to obtain the final score result.

[0057] In S2 of this invention, the specific process of identifying trajectory segments that continuously deviate from the standard path is as follows:

[0058] The collected instrument movement trajectory data is first processed using a sliding window averaging algorithm. The window size is set to a fixed value, determined based on the minimum action unit of bronchoscopy and the data sampling frequency, ensuring smoothing of high-frequency noise in the trajectory data while preserving effective operational characteristics. The trajectory data is processed segment by segment through the sliding window, and a processed trajectory point is output after each window calculation is completed, forming a complete smoothed trajectory sequence.

[0059] For the smoothed trajectory sequence, the spatial distance between each point and its corresponding point on the standard operation path is calculated. The spatial distance calculation is based on the position coordinates in a three-dimensional coordinate system. The corresponding points on the standard operation path are determined by timestamp alignment to ensure that each processed trajectory point can be matched with a reference point on the standard path at the same operation time.

[0060] Set distance thresholds for the exploration phase and the localization phase, with the exploration phase threshold being greater than the localization phase threshold. This difference in thresholds is set based on clinical operational logic. During the exploration phase, physicians need to conduct a wide-ranging exploration of the target area, allowing for greater path fluctuations; during the localization phase, physicians need to accurately reach specific anatomical locations, requiring higher path accuracy.

[0061] The calculated spatial distance sequence is then assessed for continuity. During the exploration phase, if the spatial distance between multiple consecutive trajectory points exceeds the exploration phase distance threshold, this continuous trajectory segment is marked as an exploration deviation segment. The number of consecutive points is set based on the minimum duration of the reference operation and the data sampling frequency to ensure that single distance exceedances caused by momentary erroneous operations are excluded. During the positioning phase, if the spatial distance between multiple consecutive trajectory points exceeds the positioning phase distance threshold, this continuous trajectory segment is marked as a positioning deviation segment. Finally, all marked exploration deviation segments and positioning deviation segments are merged to form a complete set of path deviation segments. During the merging process, overlapping portions between segments must be removed to ensure that each trajectory point belongs to only one deviation segment.

[0062] In S3 of this invention, the specific process of extracting the bronchial bifurcation location marker and the mucosal morphological feature marker is as follows:

[0063] Gaussian pyramid decomposition is performed on each frame of the real-time video stream. The decomposition process involves multiple Gaussian blurring and downsampling operations to generate image sequences at multiple scales. Different scales correspond to different resolutions, ensuring the detection of bronchial bifurcation structures of varying sizes. Within each scale image, a local extremum detection algorithm is used to identify potential bifurcation candidate points. Local extrema are determined by comparing the grayscale values ​​of a pixel with its neighboring pixels; points with significantly higher or lower grayscale values ​​than their neighboring pixels are marked as candidate points.

[0064] For each bifurcation candidate point, the area and contour curvature of its region are calculated. The region area is determined by counting the number of connected pixels surrounding the candidate point, and the contour curvature is calculated by fitting the edge curves of the pixels surrounding the candidate point and calculating the curvature value of the curves. Candidate points are filtered according to preset area and curvature thresholds, which are set based on the size and morphological characteristics of normal bronchial bifurcation structures. Candidate points that simultaneously meet the area and curvature threshold ranges are marked as bronchial bifurcation location identifiers.

[0065] A region growing algorithm is applied to video frame images to segment mucosal regions. The region growing algorithm uses typical gray values ​​of mucosal tissue as seed points. The growth range is determined by judging the gray similarity between adjacent pixels and seed points. The similarity judgment is based on a preset gray difference threshold. Pixels that meet the conditions are gradually merged into continuous mucosal regions, while background and other non-mucosal tissue regions are excluded.

[0066] Features are calculated for the segmented mucosal region, including gradient orientation histogram features, local binary pattern features, and mean and variance of color channels. The gradient orientation histogram is generated by calculating the gradient direction of pixels within the mucosal region, dividing the gradient direction into multiple intervals, and statistically analyzing the pixel distribution within each interval to form a histogram. The local binary pattern features are generated by comparing the grayscale values ​​of each pixel with its neighboring pixels, generating binary codes, and statistically analyzing the distribution characteristics of these codes. The mean and variance of color channels are calculated by separately calculating the pixel mean and variance of the mucosal region across the RGB color channels.

[0067] The three features mentioned above are merged into a multi-dimensional feature vector in a preset order. Normalization is then performed on the feature vector. Normalization is achieved by mapping the value of each element in the feature vector to the range of 0 to 1, eliminating the dimensional differences between different feature dimensions, and finally generating a mucosal morphological feature identifier.

[0068] In S4 of this invention, the specific process of constructing the operation logic analysis report is as follows:

[0069] First, the time windows of the operation process are divided based on the temporal distribution characteristics of the path deviation segments. The duration of the time window is set with reference to the average duration of the path deviation segments and the data sampling frequency to ensure that each time window can completely cover at least one basic operation unit, while avoiding excessively large windows that would cause feature information to overlap. The entire operation process is traversed by sliding time windows, with the window sliding step size consistent with the window duration, ensuring that the time series of the operation process is completely divided without any time segment repetition or omission, ultimately forming a continuous time window sequence.

[0070] Within each time window, the spatial distribution density of bronchial bifurcation location markers is statistically analyzed. This spatial distribution density is calculated by dividing the three-dimensional operating space corresponding to the time window into uniform cubic grid cells, counting the number of bronchial bifurcation location markers within each grid cell, and then calculating the number of markers per unit volume based on the grid cell volume. This yields the local density value for each grid cell. The local density values ​​of all grid cells together constitute the spatial distribution density matrix of bronchial bifurcation location markers within that time window.

[0071] Calculate the change in feature vectors of mucosal morphological feature identifiers between adjacent time windows. Extract multi-dimensional feature vectors corresponding to the mucosal morphological feature identifiers for each of the two adjacent time windows. Calculate the difference between the feature vector of the later time window and the feature vector of the previous time window in each dimension to obtain the feature change value of each dimension. Arrange the feature change values ​​of all dimensions in the original feature vector order to form the feature vector change of the adjacent time window pair.

[0072] A spatiotemporal correlation map between operative trajectories and anatomical structures is constructed using spatial distribution density and eigenvector changes. The node layer of the spatiotemporal correlation map includes spatial coordinates of time window markers, bronchial bifurcation location markers, and mucosal morphological feature vectors. The edge layer connects different nodes through spatial and temporal correlation edges. Spatial correlation edges are constructed based on the spatial distribution density matrix, linking bifurcation location markers within grid cells with density values ​​exceeding a set threshold to the operative trajectory segments within the corresponding time windows. Temporal correlation edges are constructed based on eigenvector changes, linking mucosal morphological feature markers within adjacent time windows with changes below a set threshold, ultimately forming a map structure that incorporates spatiotemporal correlations.

[0073] An operational logic network is generated based on the spatiotemporal correlation graph, and then converted into a structured report. The structured report includes a network node statistics table, an edge weight distribution table, and a description of spatiotemporal correlation rules. The node statistics table records the operation time, anatomical location, and feature vector information corresponding to each node. The edge weight distribution table records the connection type and weight coefficient of the edges. The description of spatiotemporal correlation rules extracts the correspondence between the operation trajectory and the changes in the anatomical structure based on the correlation relationships in the graph.

[0074] The generation process of the operation logic network is as follows:

[0075] The spatial distribution density and feature vector changes within each time window are processed sequentially to construct node feature vectors. For a single time window, the spatial distribution density matrix is ​​first expanded row-wise into a one-dimensional density vector. Then, the feature vector change corresponding to that time window (or the zero vector if it is the first time window) is used as a supplementary dimension and concatenated in the order of density vector first and change last to form the node feature vector corresponding to that time window. The node feature vectors of all time windows are arranged in chronological order to form a node set.

[0076] The cosine similarity between feature vectors of adjacent time window nodes is calculated to determine the weight coefficient of the edge. For the feature vectors of two adjacent nodes, the similarity is obtained by calculating the cosine of the angle between the vectors. The similarity value is directly used as the transition probability of the edge connecting the two nodes, i.e., the weight coefficient. If the similarity value is lower than a set threshold, the weight coefficient is set to zero, indicating that there is no effective state transition between the two nodes.

[0077] Arrange all node feature vectors into a feature matrix, and arrange all edge weights into an adjacency matrix. The feature matrix has the same number of rows as the number of nodes and the same number of columns as the dimension of the node feature vectors. Matrix elements represent the values ​​of each dimension of the corresponding node's feature vector. The adjacency matrix is ​​a square matrix with the same dimension as the number of nodes. Matrix elements represent the weights of the edges connecting two corresponding nodes, with elements without connecting edges set to zero. Finally, combine the feature matrix and the adjacency matrix to form an operational logic network that includes node states and state transition relationships.

[0078] In S5 of this invention, the specific process of generating the operation rationality evaluation vector through feature fusion is as follows:

[0079] First, the operation logic network in the operation logic analysis report is aligned with the real-time video stream data using timestamps. Each node in the operation logic network corresponds to a specific time window. The real-time video stream data records the operation process frame by frame. By extracting the timestamps of the operation logic network nodes and the acquisition timestamps of the video stream frames, a linear interpolation method is used to perform time calibration on the frame data with mismatched timestamps, ensuring that each operation logic network node can accurately match the video stream frame sequence within the corresponding time period.

[0080] Synchronous execution of spatial coordinate matching: The operation logic network node contains the three-dimensional spatial coordinates of the instrument trajectory. The real-time video stream frame obtains the mapping relationship between pixel coordinates and actual spatial coordinates through image calibration technology. Based on this mapping relationship, the pixel coordinates of the bronchial anatomy structure in the video stream frame are converted into three-dimensional spatial coordinates. Then, coordinate system unification is performed with the spatial coordinates of the instrument trajectory of the operation logic network node to ensure that the two are in the same three-dimensional spatial coordinate system, thereby achieving precise correspondence of spatial positions.

[0081] The temporal variation patterns of node feature vectors are extracted from the aligned operational logic network. All node feature vectors are extracted sequentially by time window to construct a temporal feature sequence. The temporal feature sequence is segmented using a sliding window method, and the first and second differences of the feature vectors within each segment are calculated to capture the trend and rate of change of the feature vectors over time. Simultaneously, the mean and standard deviation of the feature vectors within each segment are statistically analyzed. The trend, rate, mean, and standard deviation are combined to form a feature representation of the temporal variation pattern.

[0082] The spatial correspondence between bronchial image features and operational status nodes is calculated based on the spatial coordinate matching results. Bronchial image features, including the diameter, orientation, and mucosal texture features of the bronchial lumen, are extracted from the aligned video stream frames. The distance between the three-dimensional spatial coordinates of these image features and the spatial coordinates of the instrument trajectory of the operational status node is calculated. The number and type of image features within a preset radius around each operational status node are counted, and a spatial association table between the operational status node and the image features is established to represent the spatial correspondence in the form of an association table.

[0083] After converting the temporal variation patterns and spatial correspondences into feature vectors of a unified dimension, they are input into a gated recurrent unit for feature fusion. The gated recurrent unit includes a reset gate and an update gate. The reset gate controls the degree to which historical features are ignored; its weights are calculated using a sigmoid activation function, and historical state vectors of the temporal variation pattern features and spatial correspondence features are selectively retained according to their weights. The update gate controls the fusion ratio between the historical state vector and the current input feature vector; its weights are also calculated using a sigmoid activation function, and the historical state vector and the current input feature vector are superimposed according to their weights to generate candidate activation vectors.

[0084] The candidate activation vectors are nonlinearly transformed using the tanh activation function, and then combined with the weight vector output by the reset gate to generate the current state vector. This process is repeated to iterate through all input feature vectors, ultimately outputting a global feature sequence that fuses temporal and spatial information. This sequence preserves the coupling relationship between temporal variation patterns and spatial correlation features during the operation.

[0085] The processing procedure of the multi-granularity feature interaction layer is as follows:

[0086] The global feature sequence output from the gated recurrent unit is input into the multi-granularity feature interaction layer. First, the feature sequence is divided into multi-granularity segments by setting sliding windows of different lengths, dividing the global feature sequence into local operation fragment features and global operation sequence features. Short sliding windows correspond to local operation fragment features, capturing the feature details of a single operation action; long sliding windows correspond to global operation sequence features, reflecting the feature trend of the entire operation process.

[0087] Local operation segment features are decomposed into instrument motion features and anatomical environment features. Instrument motion features are obtained by extracting the directional changes, velocity fluctuations, and acceleration features of the instrument trajectory within the local segment; anatomical environment features are obtained by extracting the bronchial bifurcation density and mucosal morphological variability features in the corresponding video stream frames of the local segment. Global operation sequence features are decomposed into operation trajectory features and clinical scene features. Operation trajectory features are obtained by statistically analyzing the total length, number of turns, and path smoothness of the instrument trajectory within the global sequence; clinical scene features are obtained by analyzing the number of abnormal mucosal regions and secretion distribution features in the corresponding video stream frames of the global sequence.

[0088] The first correlation weight between the motion characteristics of the device and the clinical scene characteristics is calculated by calculating the similarity matrix of the two types of feature vectors and counting the proportion of elements in the similarity matrix that are greater than a set threshold. This proportion is the first correlation weight. The second correlation weight is obtained by calculating the similarity matrix and the proportion of elements between the anatomical environment characteristics and the operation trajectory characteristics using the same method.

[0089] The fusion ratio of local manipulation fragment features and global manipulation sequence features is determined based on the first and second correlation weights. When the first correlation weight is high, the proportion of instrument motion features in the fusion result is increased; when the second correlation weight is high, the proportion of anatomical environment features in the fusion result is increased. The local manipulation fragment features and global manipulation sequence features are weighted and summed according to the determined fusion ratio to output a multi-granularity fused feature vector.

[0090] The specific process of generating the operation rationality evaluation vector based on the output of the graph neural network is as follows:

[0091] Multi-granularity fused feature vectors are input into a graph neural network for feature aggregation. The graph neural network uses nodes of the operational logic network as graph nodes and edges as graph edges. The multi-granularity fused feature vectors are assigned to the corresponding graph nodes. Graph convolution operations are used to calculate the weighted sum of features from the neighboring nodes of each node. The weights are determined by the edge weight matrix of the operational logic network, realizing the propagation and aggregation of node features within the graph structure. Multiple graph convolution operations are repeated to gradually fuse global node feature information, ultimately outputting the aggregated node feature vectors and the updated edge weight matrix.

[0092] Read the node feature vectors and edge weight matrices output by the graph neural network, and perform temporal pooling on the node feature vectors. Divide the node feature vectors into multiple time segments according to the time window order, and perform max pooling and average pooling on the node feature vectors in each time segment. Concatenate the pooling results to form the temporal feature representation of that time segment. The temporal feature representations of all time segments together constitute the global temporal feature representation.

[0093] Spatial dimension pooling is performed on the edge weight matrix, dividing the edge weight matrix into multiple sub-matrices according to spatial regions. Summation pooling and variance pooling are performed on the weight values ​​in each sub-matrix. The pooling results are concatenated to form the spatial relationship representation of the spatial region. The spatial relationship representations of all spatial regions together constitute the global spatial relationship representation.

[0094] The global temporal feature representation and the global spatial relationship representation are concatenated to form a joint feature vector. This joint feature vector is then input into a fully connected layer for dimensionality reduction. The fully connected layer progressively reduces the feature dimension through multiple layers of neural networks until a feature vector of the preset dimension is output. The dimensionality-reduced feature vector is then input into a softmax function, which maps each element of the feature vector to a probability value between 0 and 1, with the sum of all probability values ​​being 1, forming a probability distribution vector. This probability distribution vector is the operation rationality evaluation vector.

[0095] In S6 of this invention, the specific process by which the rationality assessment vector of the analytical operation is used to obtain the path compliance score, clinical decision-making score, and adaptability score is as follows:

[0096] The three-dimensional trajectory coordinate sequence is extracted from the operational rationality evaluation vector. This sequence contains the three-dimensional spatial coordinates of the instrument at each time window during the operation. Based on this sequence, a trajectory smoothness index is calculated. By traversing adjacent coordinate points, the frequency and magnitude of trajectory direction changes are statistically analyzed. The magnitudes of these direction changes are weighted and summed, with the weights increasing with the frequency of change. The summation result is then normalized to obtain the trajectory smoothness index. When calculating the path efficiency coefficient, the straight-line distance between the operation start point and the target end point is first determined. Then, the total length of the actual trajectory is calculated. The path efficiency coefficient is obtained by converting the ratio of the total actual trajectory length to the straight-line distance.

[0097] The scenario understanding features in the operational rationality assessment vector are analyzed. These features include bronchial anatomical structure recognition information and clinical operation protocol matching information. The probability distributions related to anatomical structures in the features are extracted, and the number of matches between identified bronchial bifurcations, lumens, and other structures and standard anatomical models is statistically analyzed. Combined with the confidence values ​​of the matched structures, the accuracy of anatomical structure recognition is calculated. Simultaneously, parameters related to the clinical protocol are extracted from the features, and the degree of conformity between the actual operation steps and the standard clinical protocol is compared. The number of conforming steps and the conformity of key steps are statistically analyzed to generate the clinical protocol matching degree.

[0098] The temporal response features in the operational rationality evaluation vector are analyzed. These features record the response time and handling actions for abnormal situations during the operation. The timestamps of the abnormal situation trigger and the first response action are extracted from these features, and the time difference between them is calculated to obtain the abnormal situation detection delay. After an abnormal situation is triggered, the completion time of the operational correction action and the degree to which the correction effect meets the standard are statistically analyzed. Combined with the consistency of the correction action with the standard processing flow, the timeliness of operational correction is generated.

[0099] A path compliance score is generated based on the trajectory smoothness index and path efficiency coefficient. The two indicators are weighted and summed according to a preset ratio, with the smoothness index weight emphasizing trajectory stability and the path efficiency coefficient weight emphasizing operational economy. A clinical decision-making score is generated based on the accuracy of anatomical structure recognition and the matching degree of clinical protocols. These two indicators are fused with equal weights to comprehensively reflect the understanding of the clinical scenario and the rationality of the decision. A resilience score is generated based on the anomaly detection delay and the timeliness of operational correction. The detection delay weight emphasizes response speed, and the timeliness of correction weight emphasizes processing effectiveness. The weighted sum yields the resilience score.

[0100] In S6 of this invention, the specific process for obtaining the final scoring result is as follows:

[0101] Weighted coefficients are assigned based on the training stage. In the initial training stage, the focus is on basic operational standardization, with path adherence having the highest weight, followed by clinical decision-making, and then adaptability. In the advanced training stage, the emphasis is on clinical practice skills, with adaptability having the highest weight, followed by clinical decision-making, and then path adherence. These weights are preset through an expert system to ensure they align with the competency development objectives of each training stage.

[0102] The path adherence score, clinical decision-making score, and adaptability score were normalized, mapping each score to a numerical range of 0 to 1. A linear scaling method was used, with the theoretical maximum and minimum values ​​of each score as a benchmark, to eliminate dimensional differences between different scoring dimensions. The normalized three scores were then weighted and summed according to the weighting coefficients corresponding to the training phase to obtain preliminary scoring results.

[0103] The preliminary scoring results are mapped to a preset scoring range, which is set according to the training evaluation standards. Through linear transformation, the numerical range of the preliminary scoring results is mapped to the scoring range to ensure that the final scoring results meet the numerical requirements of the evaluation system and generate the final scoring results.

[0104] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A multifunctional bronchoscopy simulation training method, characterized in that, Includes the following steps: S1. Collect instrument movement trajectory data during bronchoscopy operation, as well as real-time video stream data of the bronchial wall. S2. Compare the instrument movement trajectory data with the standard operating path point by point, and identify the trajectory segments that continuously deviate from the standard path as path deviation segments. S3. Extract bronchial bifurcation location identifiers and mucosal morphological feature identifiers from real-time video stream data; S4. Spatiotemporally correlate the path deviation sections with the corresponding bronchial bifurcation location markers and mucosal morphological feature markers to construct an operational logic analysis report; S5. Convert the operation logic analysis report into semantic features and image features respectively, input them into the multimodal large model, and generate an operation rationality evaluation vector through feature fusion; S6. The rationality assessment vector of the operation is analyzed to obtain the path compliance score, clinical decision-making score and adaptability score. The three scores are then combined to obtain the final score result. In step S5, the specific process of generating the operation rationality evaluation vector through feature fusion is as follows: The operation logic network in the operation logic analysis report is time-stamp aligned and spatial coordinate matched with the real-time video stream data; the temporal change patterns of node feature vectors are extracted from the aligned operation logic network. The spatial correspondence between bronchial image features and operational status nodes is calculated based on the spatial coordinate matching results. The temporal variation pattern and spatial correspondence are input into the gated loop unit for feature fusion; The output of the gated recurrent unit is input into the multi-granularity feature interaction layer, which processes both local operation fragment features and global operation sequence features simultaneously. The output of the multi-granularity feature interaction layer is input into the graph neural network for feature aggregation, and an operation rationality evaluation vector is generated based on the output of the graph neural network. The processing procedure of the multi-granularity feature interaction layer is as follows: Local operation segment features are decomposed into instrument motion features and anatomical environment features, and global operation sequence features are decomposed into operation trajectory features and clinical scene features; Calculate the first correlation weight between instrument motion characteristics and clinical scenario characteristics, and calculate the second correlation weight between anatomical environment characteristics and operation trajectory characteristics; The fusion ratio of local operation fragment features and global operation sequence features is determined based on the first correlation weight and the second correlation weight. Weighted fusion outputs a multi-granularity fused feature vector; The specific process of generating the operation rationality evaluation vector based on the output of the graph neural network is as follows: Read the node feature vectors and edge weight matrices of the graph neural network, perform temporal pooling on the node feature vectors to extract temporal feature representations, and perform spatial pooling on the edge weight matrices to extract spatial relation representations; The feature representations of the time and space dimensions are concatenated, and the concatenated feature representations are input into a fully connected layer for dimensionality reduction. The dimensionality-reduced features are mapped to a probability distribution through the softmax function, and the probability distribution vector is output as the operation rationality evaluation vector.

2. The multifunctional bronchoscopy simulation training method according to claim 1, characterized in that, In step S2, the specific process of identifying trajectory segments that continuously deviate from the standard path is as follows: A sliding window averaging algorithm is applied to the instrument movement trajectory data, with the window size set to a fixed value, to calculate the spatial distance between the processed trajectory points and the corresponding points of the standard operating path. Set a distance threshold for the exploration phase and a distance threshold for the positioning phase, wherein the distance threshold for the exploration phase is greater than the distance threshold for the positioning phase; Determine the spatial distance sequence of continuous trajectory points. When the spatial distance between multiple consecutive points in the exploration phase exceeds the distance threshold of the exploration phase, it is marked as an exploration deviation segment. When the spatial distance between multiple consecutive points in the positioning phase exceeds the positioning phase distance threshold, it is marked as a positioning deviation segment; The exploration deviation section and the positioning deviation section will be merged into the path deviation section.

3. The multifunctional bronchoscopy simulation training method according to claim 1, characterized in that, In step S3, the specific process of extracting the bronchial bifurcation location markers and mucosal morphological feature markers is as follows: Gaussian pyramid decomposition is performed on video frames to generate images at multiple scales. Local extrema are detected in each scale image and used as candidate bifurcation points. The area and contour curvature of the region where the candidate bifurcation points are located are calculated. Candidate points are filtered according to area thresholds and curvature thresholds. Candidate points that meet the threshold requirements are marked as bronchial bifurcation location markers. The mucosal region is segmented by performing a region growing algorithm on the video frame. The set features of the mucosal region are calculated, including gradient direction histogram features, local binary pattern features, and mean and variance of color channels. The set features are merged into feature vectors and normalized to generate mucosal morphology feature identifiers.

4. The multifunctional bronchoscopy simulation training method according to claim 1, characterized in that, In step S4, the specific process of constructing the operation logic analysis report is as follows: Based on the temporal distribution characteristics of the path deviation section, the operation process is divided into multiple time windows. Within each time window, the spatial distribution density of the bronchial bifurcation location markers is statistically analyzed, and the change in the feature vector of the mucosal morphological feature markers between adjacent time windows is calculated. A spatiotemporal correlation map between the operation trajectory and the anatomical structure is established using spatial distribution density and eigenvector changes. An operation logic network is generated based on the spatiotemporal correlation map. The operation logic network contains a set of nodes and a set of edges. Nodes represent operation states, and edges represent state transitions. The operation logic network is then converted into a structured operation logic analysis report.

5. The multifunctional bronchoscopy simulation training method according to claim 4, characterized in that, The generation process of the operation logic network is as follows: The spatial distribution density and feature vector change within each time window are processed sequentially, and the spatial distribution density and feature vector change are combined into node feature vectors. Calculate the cosine similarity between the feature vectors of adjacent time window nodes, determine the transition probability between nodes based on the cosine similarity value, and assign the transition probability value as the weight coefficient of the connecting edge; Arrange all node feature vectors into a feature matrix, arrange all edge weights into an adjacency matrix, and combine the feature matrix and adjacency matrix to generate the operational logic network.

6. The multifunctional bronchoscopy simulation training method according to claim 1, characterized in that, In step S6, the specific process by which the rationality assessment vector of the analytical operation is used to obtain the path compliance score, clinical decision-making score, and adaptability score is as follows: Extract the three-dimensional trajectory coordinate sequence from the operation rationality assessment vector, and calculate the trajectory smoothness index and path efficiency coefficient; analyze the scene understanding features in the operation rationality assessment vector to generate the anatomical structure recognition accuracy and clinical protocol matching degree; analyze the temporal response features in the operation rationality assessment vector to calculate the abnormal situation detection delay and operation correction timeliness. A path compliance score is generated based on the trajectory smoothness index and the path efficiency coefficient. Clinical decision scores are generated based on the accuracy of anatomical structure recognition and the matching degree of clinical protocols. A response capability score is generated based on the delay in anomaly detection and the timeliness of operational correction.

7. The multifunctional bronchoscopy simulation training method according to claim 6, characterized in that, In step S6, the specific process for obtaining the final score result is as follows: Weighting coefficients are set according to the training stage. In the primary training stage, the weighting order is path compliance score, clinical decision score, and adaptability score. In the advanced training stage, the weighting order is adaptability score, clinical decision score, and path compliance score. The path adherence score, clinical decision-making score, and adaptability score were normalized. The three scores were then weighted and summed according to the weighting coefficients of the corresponding training stages. The weighted sum was then mapped to the score interval to generate the final score.

Images