A text-guided intelligent evaluation method for clinical skills based on ranking graph constraints

Abstract

The application relates to the technical field of intelligent evaluation of medical education, in particular to a text-guided intelligent evaluation method for clinical skills based on a ranking graph constraint. The method comprises the following steps: segmenting and extracting features from a clinical skill video, pre-processing the video, combining video time sequence information and action speed text embedding to construct a time sequence feature representation, introducing skill classification text and a learnable memory embedding vector, realizing multi-modal feature decoding based on a cross-attention mechanism, outputting a skill score prediction value through a scoring network, constructing a ranking relationship matrix by using real score differences, introducing a ranking graph constraint loss and a scoring loss for joint optimization, and thus improving the accuracy and ranking consistency of skill evaluation. The application realizes accurate modeling of a clinical skill operation process and effective discrimination of relative levels between samples, thereby significantly improving the accuracy, consistency and reliability of skill evaluation.

Description

Technical Field

[0001] This application relates to the field of intelligent evaluation technology in medical education, and in particular to a text-guided intelligent evaluation method for clinical skills based on ordination graph constraints. Background Technology

[0002] In related technologies, with the continuous improvement of the informatization and intelligentization of medical education, video-based clinical skills assessment has become an important technical means for clinical nursing training and skills evaluation. This type of method typically uses videos of clinical skills operations as input, extracts features and performs temporal modeling of the operation process in the video using computer vision and deep learning models, and combines these with a scoring model to quantitatively assess the standardization, proficiency, and quality of completion of the operation. This achieves automated evaluation of clinical skills, helps reduce the cost of manual scoring, and improves the consistency and objectivity of evaluation results.

[0003] However, existing clinical skills video assessment methods still face significant performance and modeling challenges in practical applications. On the one hand, some methods employ pairwise comparison-based ranking learning strategies, constructing video pairs to characterize skill superiority / inferiority relationships. While this can improve the model's discriminative ability, the number of paired samples increases exponentially with the sample size, leading to a significant increase in training complexity and computational cost, making it unsuitable for large-scale data scenarios. On the other hand, although some methods introduce textual descriptions or multimodal information to enhance the understanding of skill semantics, they mostly employ simple feature concatenation or shallow fusion methods, lacking effective modeling of the deep correlation between textual semantics and video temporal features, making it difficult to fully leverage the guiding role of semantic information in skills assessment. Furthermore, most existing methods rely on single videos for score prediction, failing to fully utilize the overall ranking relationships between different videos, resulting in insufficient consistency and stability of scoring results globally. In addition, existing technologies lack effective prior modeling mechanisms for the differences in the rhythm and speed of action execution, which are crucial for representational clinical skills, making it difficult to finely distinguish skill levels under different operational rhythms.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0005] The main objective of this application is to propose a text-guided intelligent evaluation method for clinical skills based on ordination graph constraints, so as to achieve accurate modeling of the clinical skills operation process and effective discrimination of the relative levels between samples, thereby significantly improving the accuracy, consistency and reliability of skills evaluation.

[0006] To achieve the above objectives, one aspect of this application proposes a text-guided intelligent assessment method for clinical skills based on ordination graph constraints, the method comprising the following steps: The input clinical skills video is subjected to frame sampling and segmentation processing to extract segment-level visual feature sequences; The segment-level visual feature sequence is mapped and normalized to obtain a preprocessed video feature representation; The videos are sorted and grouped based on their duration, and the preset motion speed description text is encoded to obtain the motion speed text embedding. Generate video temporal embeddings based on the temporal order of video segments; An action-time embedding is constructed based on the video temporal embedding and the action speed text embedding; The action timing embedding is fused with the preprocessed video feature representation to obtain a timing feature representation; The preset skill level description text is encoded to obtain the skill level text embedding, and a learnable memory embedding vector is introduced. The temporal feature representation is decoded based on the cross-attention mechanism to obtain the decoded feature representation. The decoded feature representation is input into the scoring network to obtain the predicted skill score; A ranking relation matrix is ​​constructed based on the actual skill score differences, and a ranking graph constraint loss is calculated based on the ranking relation matrix and the predicted skill score values. The ranking graph constraint loss and the score loss are then jointly optimized.

[0007] In some embodiments, the step of performing frame sampling and segmentation processing on the input clinical skills video to extract segment-level visual feature sequences includes: The input clinical skills video is uniformly sampled according to a preset number of sampling frames to obtain a video frame sequence; The video frame sequence is segmented based on a preset number of frames per segment. When the number of frames is insufficient, frame interpolation is performed to fill in the gaps, resulting in multiple video segments. Each video segment is input into a dilated 3D convolutional network for encoding to obtain the corresponding segment-level visual features; The segment-level visual features are combined in chronological order to obtain a segment-level visual feature sequence.

[0008] In some embodiments, the step of sorting and grouping videos based on video duration and encoding preset motion speed description text to obtain motion speed text embedding includes: Multiple sets of action speed description texts are preset to characterize the differences in the execution speed of clinical skills, and the action speed description texts are encoded to obtain the corresponding action speed text embeddings; The clinical skills videos are sorted based on their duration and divided into multiple action speed groups according to preset grouping rules. Based on the multiple action speed groups, corresponding action speed text embeddings are assigned to each clinical skill video, and the action speed text embeddings are used as semantic prior information of action speed.

[0009] In some embodiments, generating video temporal embeddings based on the temporal order of video segments includes: Based on the temporal order of video segments in the clinical skills videos, determine the time index position corresponding to each video segment; Based on the time index position, a corresponding temporal embedding representation is generated for each video segment; The temporal embedding representation is combined with a preset time position encoding function to obtain video temporal embedding.

[0010] In some embodiments, the formula for the time position encoding function is as follows: ; in, Indicates the first Embed the time location corresponding to each clinical skills video; Indicates an embedded dimension index, when When the number is even, the sine function is used. When the number is odd, the cosine function is used; Indicates the feature embedding dimension; Indicates the first Temporal location-related features of each clinical skills video; Indicates the first The overall time rhythm or time scale parameter of a clinical skills video.

[0011] In some embodiments, fusing the action timing embedding with the preprocessed video feature representation to obtain a timing feature representation includes: The action timing embedding is dimensionally aligned with the preprocessed video feature representation; The dimension-aligned action temporal embedding and the preprocessed video feature representation are fused element-wise to obtain the fused feature representation corresponding to each video segment; The fused feature representation is subjected to linear transformation and normalization to obtain the temporal feature representation.

[0012] In some embodiments, encoding the preset skill level description text to obtain a skill level text embedding, introducing a learnable memory embedding vector, and decoding the temporal feature representation based on a cross-attention mechanism to obtain a decoded feature representation includes: The preset skill level description text is encoded to obtain the skill level text embedding; A learnable memory embedding vector is introduced, and a query vector is constructed based on the skill-level text embedding. Using the query vector as the query and the temporal feature representation as the key and value, the learnable memory embedding vector is updated through a cross-attention mechanism to obtain the memory feature representation; Using the memory feature representation as the query and the temporal feature representation as the key and value, the temporal feature representation is decoded through a cross-attention mechanism to obtain the decoded feature representation.

[0013] In some embodiments, inputting the decoded feature representation into the scoring network to obtain a predicted skill score includes: The decoded feature representation is input into the scoring network, and feature mapping is performed through a multi-layer fully connected network to obtain the scoring feature vector; Based on the rating feature vector, the skill rating prediction value is obtained by mapping through an activation function.

[0014] In some embodiments, the step of constructing a ranking relation matrix based on the actual skill score differences, calculating the ranking graph constraint loss based on the ranking relation matrix and the predicted skill score values, and jointly optimizing the ranking graph constraint loss and the score loss includes: Construct a real ranking relationship matrix based on the differences in real skill scores of clinical skill videos within the same training batch; Construct a prediction ranking matrix based on the predicted skill scores; The ranking graph constraint loss is calculated based on the difference between the actual ranking relation matrix and the predicted ranking relation matrix; The sorting graph constraint loss and the scoring loss are weighted and fused to obtain the joint loss function; The parameters are optimized based on the joint loss function.

[0015] In some embodiments, the formula for calculating the ordering graph constraint loss is as follows: ; in, This represents the loss due to sorting graph constraints. This represents the number of clinical skills videos participating in the ranking constraint; i and j represent the video sample indices. Represents the weights or mask matrix elements of the sorting constraints; This represents the elements in the prediction ranking relation matrix; This represents an element in the actual sorting matrix.

[0016] To achieve the above objectives, another aspect of this application proposes a text-guided intelligent assessment system for clinical skills based on ordination graph constraints, the system comprising: The first module is used to perform frame sampling and segmentation processing on the input clinical skills video, and extract segment-level visual feature sequences; The second module is used to map and normalize the segment-level visual feature sequence to obtain a preprocessed video feature representation. The third module is used to sort and group videos based on video duration and encode preset motion speed description text to obtain motion speed text embedding. The fourth module is used to generate video temporal embeddings based on the temporal order of video segments; The fifth module is used to construct an action temporal embedding based on the video temporal embedding and the action speed text embedding; The sixth module is used to fuse the action timing embedding with the preprocessed video feature representation to obtain a timing feature representation; The seventh module is used to encode the preset skill level description text to obtain the skill level text embedding, and introduce a learnable and memory-based embedding vector. Based on the cross-attention mechanism, the temporal feature representation is decoded to obtain the decoded feature representation. The eighth module is used to input the decoded feature representation into the scoring network to obtain the predicted skill score value; The ninth module is used to construct a ranking relation matrix based on the actual skill score differences, calculate the ranking graph constraint loss based on the ranking relation matrix and the predicted skill score values, and jointly optimize the ranking graph constraint loss and the score loss.

[0017] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0018] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0019] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0020] The embodiments of this application include at least the following beneficial effects: This application provides a text-guided intelligent evaluation method for clinical skills based on ordination graph constraints. This scheme extracts stable and discriminative segment-level visual features by performing frame sampling, segmentation, and feature mapping on clinical skills videos. It then groups these features by combining them with video duration for ordination. Furthermore, it introduces text embedding of action speed descriptions and video temporal embeddings for joint modeling, thereby effectively characterizing the rhythm and temporal evolution of actions during clinical operations and improving the ability to recognize operational standardization and coherence. Further, by fusing action temporal embeddings with video feature representations and combining skill-level text embeddings, learnable memory embedding vectors, and cross-attention mechanisms for decoding, it achieves deep alignment between video process information and scoring semantics. This enables precise discrimination of key operational segments for different skill levels, thereby improving the accuracy and interpretability of scoring results. Simultaneously, it constructs an ordination relationship matrix based on real scoring differences and jointly optimizes ordination graph constraint loss and scoring loss. While ensuring the accuracy of scoring regression, it strengthens the relative ordination relationship between samples, effectively avoiding inconsistencies in scoring results and improving the stability and consistency of the overall evaluation. This ultimately achieves a more objective, refined, and reliable intelligent evaluation of clinical skill levels. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a text-guided intelligent assessment method for clinical skills based on ordination graph constraints provided in an embodiment of this application. Figure 2 yes Figure 1 A flowchart illustrating step S3 in the process; Figure 3 yes Figure 1 A flowchart illustrating step S4 in the process; Figure 4 yes Figure 1 A flowchart illustrating step S7 in the process; Figure 5 yes Figure 1 A flowchart illustrating step S8 in the process; Figure 6 yes Figure 1 The flowchart of step S9 is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0023] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0024] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0026] Before providing a detailed description of the embodiments of this application, some of the nouns and terms used in the embodiments of this application will be explained first. The nouns and terms used in the embodiments of this application shall be interpreted as follows: Clinical skill video refers to continuous video data that records the process of a subject performing a clinical procedure. It is used to characterize information such as the procedure, the standardization of the action, and the quality of the performance. It is the basic data carrier for the skills assessment in this application. Segment-level visual features refer to feature vectors that represent local spatiotemporal information extracted from each video segment by a deep learning model after segmenting clinical skills videos. These features are used to describe the action state and visual content within that time segment. Inflated 3D Convolutional Network (I3D Backbone Network) is a deep neural network structure that expands two-dimensional convolutional kernels to three-dimensional space to simultaneously model spatial features and temporal dynamic information in video data, thereby enabling the extraction of spatiotemporal features from video clips. The temporal positional encoding function is a function used to map time index information into a continuous vector representation. It is usually built based on sine and cosine functions to enhance the model's ability to perceive sequence order information. Cross attention mechanism is an attention mechanism based on the correlation between query, key and value, used to realize information interaction and fusion between features of different modalities. In this application, it is used for alignment modeling of text semantics and video features. A learnable memory embedding vector is a memory vector obtained through parameter optimization during model training. It is used to store semantic information related to skill levels and participates in feature updates and expression enhancement during decoding. Decoded feature representation refers to the feature representation obtained by semantically decoding the fused temporal features through a cross-attention mechanism, which is used to reflect the relationship between video content and skill level semantics. A scoring network is a regression model built on a neural network structure, used to map decoded features to corresponding predicted skill scores. Ranking Graph Constraint Loss is a loss function constructed based on the difference between the true ranking matrix and the predicted ranking matrix, used to constrain the model output to meet the ranking consistency among samples. The joint loss function is a comprehensive loss function obtained by weighted fusion of the ranking graph constraint loss and the scoring loss. It is used to improve ranking consistency while ensuring scoring accuracy.

[0027] This application provides a text-guided intelligent evaluation method for clinical skills based on ordination graph constraints. This method extracts stable and discriminative segment-level visual features from clinical skills videos through frame sampling, segmentation, and feature mapping. These features are then grouped and ranked according to video duration. Joint modeling is performed using text embeddings describing action speed and video temporal embeddings to effectively characterize the rhythm and temporal evolution of actions during clinical procedures, improving the ability to recognize operational standardization and coherence. Furthermore, by fusing action temporal embeddings with video feature representations and combining skill-level text embeddings, learnable memory embedding vectors, and cross-attention mechanisms for decoding, deep alignment between video process information and scoring semantics is achieved. This enables precise discrimination of key operational segments for different skill levels, thereby improving the accuracy and interpretability of the scoring results. Simultaneously, an ordination relationship matrix based on real scoring differences is constructed, and ordination graph constraint loss and scoring loss are jointly optimized. This ensures scoring regression accuracy while strengthening the relative ordination relationship between samples, effectively avoiding inconsistencies in scoring results and improving the overall stability and consistency of the evaluation. Ultimately, this achieves a more objective, refined, and reliable intelligent evaluation of clinical skill levels.

[0028] This application provides a text-guided intelligent assessment method for clinical skills based on ordination graph constraints, relating to the field of intelligent assessment technology in medical education. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing a text-guided intelligent assessment method for clinical skills based on ordination graph constraints, but is not limited to the above forms.

[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0030] Figure 1 This is an optional flowchart of a text-guided intelligent assessment method for clinical skills based on ordination graph constraints provided in this application embodiment. Figure 1 The method may include, but is not limited to, steps S1 to S9: S1: Perform frame sampling and segmentation on the input clinical skills video to extract segment-level visual feature sequences; The process involves frame sampling and segmentation of the input clinical skills videos to extract segment-level visual feature sequences, including: S11: The input clinical skills video is uniformly sampled according to a preset number of sampling frames to obtain a video frame sequence; S12: Segment the video frame sequence based on a preset number of frames per segment. When the number of frames is insufficient, insert frames to complete the segment and obtain multiple video clips. S13: Input each video segment into a dilated 3D convolutional network for encoding to obtain the corresponding segment-level visual features; The segment-level visual features are combined in chronological order to obtain a segment-level visual feature sequence.

[0031] In this embodiment, operation videos from clinical skills assessment scenarios are collected as input samples. The videos can be recorded by a fixed-position camera with a resolution of, for example, 1920×1080 and a frame rate of, for example, 25fps or 30fps. Each video corresponds to the entire process of a subject completing a specific clinical skill operation, and a professional instructor or examiner provides a real skill score according to a unified scoring standard, serving as supervisory information during the model training phase. To facilitate model training, all videos can be organized into training, validation, and test sets, with standardized video formats, duration recording, and scoring annotation methods.

[0032] For a single input clinical skill video, uniform sampling is first performed according to a preset number of frames. For example, several frames are extracted from the original video at fixed time intervals to obtain a sampled video frame sequence. If the preset total number of sampled frames is 96, with each segment containing 16 frames, the video can be divided into 6 video segments of equal length. If the number of sampled frames is not divisible by the number of frames in each segment, uniform frame interpolation is used to pad the end segments to ensure that all video segments are of consistent length. This processing method can preserve the complete temporal information of the operation process as much as possible while unifying the scale of subsequent network inputs.

[0033] After video segmentation, each video segment is sequentially input into a dilated 3D convolutional network (I3D backbone network) for feature extraction. The I3D backbone network can simultaneously capture spatial appearance information and short-term dynamic motion information, thereby extracting segment-level visual features corresponding to each video segment. For clinical skills operations, these segment-level visual features can characterize the operator's hand movements, instrument usage status, body posture changes, and interaction with the object being operated on at a certain stage, such as whether the disinfection action is complete, whether the instrument holding action is standardized, and whether the operation rhythm is stable. Arranging the output features of each segment in chronological order yields the segment-level visual feature sequence corresponding to that video.

[0034] For example, in an intravenous infusion skills assessment scenario, the first two video segments might primarily correspond to the instrument preparation stage, several segments in the middle to the puncture and fixation stage, and the final segment to the cleanup and completion stage. Through the above processing, the original video is transformed into a segment-level visual feature sequence arranged in chronological order, laying the foundation for subsequent prior modeling of action speed and semantic decoding of skills.

[0035] Specifically, input clinical skills videos as The number of original frames after sampling is Each video segment contains 100 frames. .when Cannot be During the division process, the total number of video frames is padded to the nearest whole number by uniformly interpolating the video frames. Then, the completed video frames were divided into time sequences. Each video segment is of uniform length, and the segments are input into the I3D backbone network for feature encoding to obtain the corresponding segment-level visual feature representations.

[0036] S2: Map and normalize the segment-level visual feature sequence to obtain the preprocessed video feature representation; In this embodiment, since the original segment-level visual features output by the I3D backbone network are typically high-dimensional, and the feature distributions between different video samples may have scale differences, the segment-level visual features obtained in step S1 are first linearly mapped. Specifically, a fully connected mapping layer can be set to uniformly map the original features to a preset feature space, for example, mapping from the original dimension d1 to the target dimension d. This ensures that the visual feature dimension is consistent with the dimensions of subsequent text embedding and temporal embedding, facilitating the calculation of summation, fusion, and cross-attention.

[0037] After linear mapping, the mapping result is normalized to make the features of different segments more stable in numerical range and reduce gradient fluctuations during training. After processing, a preprocessed video feature representation with uniform dimensions and stable values ​​can be obtained. This preprocessed video feature not only preserves the visual dynamic information in the original clinical skill operation, but also facilitates element-wise fusion with the subsequently constructed action temporal embedding.

[0038] Specifically, the segment-level visual features output by the I3D backbone network are represented as follows: ,in, Indicates the number of video clips. This represents the original feature dimension. A linear mapping operation is used to map segment-level visual features to a predefined feature dimension space. The mapping results are then normalized to obtain the preprocessed video features. These serve as input features for subsequent action temporal encoding and skill semantic decoding modules. and These represent the weight matrix and bias parameters of the linear mapping, respectively.

[0039] S3: Sort and group the videos based on their duration, and encode the preset motion speed description text to obtain the motion speed text embedding; Among them, reference Figure 2 As shown, the videos are sorted and grouped based on their duration, and the preset motion speed description text is encoded to obtain the motion speed text embedding, including: S31: Pre-set multiple sets of action speed description texts to characterize the differences in the execution speed of clinical skills, and encode the action speed description texts to obtain the corresponding action speed text embeddings; S32: Sort the clinical skills videos based on their duration and divide them into multiple action speed groups according to preset grouping rules; S33: Based on multiple motion speed groups, assign corresponding motion speed text embeddings to each clinical skill video, and use the motion speed text embeddings as semantic prior information of motion speed.

[0040] In this embodiment, multiple sets of descriptive texts for action speed are first pre-set to characterize differences in the execution speed of clinical skills. These descriptive texts are used to depict the clinical skill performance under different operational rhythms, and may include semantic descriptions at different levels such as "faster action execution, tighter rhythm," "moderate action execution, smoother process," and "slower action execution, with pauses or hesitations." Subsequently, each set of descriptive texts is encoded to convert the text information into corresponding action speed text embeddings, giving them semantic representation capabilities that can participate in subsequent feature modeling, and serving as semantic vectors reflecting the characteristics of operational rhythm.

[0041] Specifically, the pre-trained S3D text encoder is used to encode the text describing the action speed, thereby obtaining the corresponding action speed text embedding features. ,in, Indicates the first Group action speed description text.

[0042] Furthermore, all video samples are sorted based on the duration of the clinical skills videos and divided into multiple action speed groups according to preset grouping rules. Specifically, all videos are sorted from shortest to longest duration and divided into several groups according to proportion or interval. Videos with shorter durations are classified as faster action groups, videos with medium durations as moderate action groups, and videos with longer durations as slower action groups. In this way, continuously changing operation durations are mapped to action speed categories with clear semantic meanings, thereby achieving structured modeling of the execution rhythm of clinical skills.

[0043] After grouping the videos, each clinical skill video is assigned a corresponding action speed text embedding based on its action speed group. This text embedding is then used as semantic prior information for the video's action speed and input into the subsequent model. By introducing this semantic prior information, the model can simultaneously perceive the differences in operational content and execution rhythm when modeling the temporal features of the videos. This improves the model's ability to distinguish the quality of clinical skill operations and provides richer and more discriminative semantic support for subsequent action temporal embedding construction and skill score prediction.

[0044] S4: Generate video temporal embeddings based on the temporal order of video segments; Among them, reference Figure 3 As shown, video temporal embedding is generated based on the temporal order of video segments, including: S41: Determine the time index position of each video segment based on the temporal order of the video segments in the clinical skills video. S42: Generate corresponding temporal embedding representations for each video segment based on the time index position; S43: Combine the temporal embedding representation with the preset time position encoding function to obtain the video temporal embedding.

[0045] In this embodiment, for the multiple video segments formed after segmenting a clinical skills video, the temporal index position of each video segment is first determined according to its arrangement order within the entire video. Specifically, each video segment is assigned a corresponding temporal position identifier according to its sequence within the original clinical skills operation flow, with the first video segment corresponding to the start stage, the middle video segment to the execution stage, and the last video segment to the end stage. This method clearly distinguishes the stage position of each video segment in the clinical skills operation process, preventing subsequent models from relying solely on local visual content and ignoring the inherent temporal sequence of the operation flow itself.

[0046] Furthermore, based on the time index position, corresponding temporal embedding representations are generated for each video segment. These temporal embedding representations characterize the positional information of each video segment in the time dimension, enabling video segments at different time positions to have distinguishable temporal semantics in the feature space. For clinical skills operation videos, even if different video segments have certain similarities in visual appearance, by introducing corresponding temporal embedding representations, the model can still distinguish whether the segment is in the preparation stage, the key operation stage, or the finishing stage, thereby more accurately depicting the stage-change characteristics in the clinical skills operation process.

[0047] After generating temporal embedding representations for each video segment, these representations are combined with a predefined temporal position encoding function to obtain the video temporal embedding. This video temporal embedding not only preserves the sequential information of the video segments but also enhances the structured representation between different time positions, enabling the model to simultaneously perceive the visual content and temporal position differences of video segments during subsequent action temporal feature construction. For example, in clinical nursing procedures, the same hand movements appearing in the early and late stages of an operation typically have different skill meanings. By introducing video temporal embedding, the model's ability to represent the completeness of the operational process, the rationality of the stages, and the accuracy of the action sequence can be effectively improved, providing temporal information support for subsequent action temporal embedding construction and skill score prediction.

[0048] Specifically, the time-position encoding function is constructed based on the traditional Transformer position encoding method, and its expression is as follows: ; in, Indicates the first Embed the time location corresponding to each clinical skills video; Indicates an embedded dimension index, when When the number is even, the sine function is used. When the number is odd, the cosine function is used; Indicates the feature embedding dimension; Indicates the first Temporal location-related features of each clinical skills video; Indicates the first The overall time rhythm or time scale parameter of a clinical skills video.

[0049] By using the aforementioned time-location encoding method, the location information of video segments in the time dimension is explicitly introduced into the subsequent action sequence feature modeling and text-guided memory decoding process.

[0050] S5: Constructing action temporal embedding based on video temporal embedding and action speed text embedding; In this embodiment, after obtaining the motion speed text embedding and the video temporal embedding, the two are added element-wise to fuse motion speed semantics and temporal position semantics. Since the motion speed text embedding reflects the overall operation rhythm, while the temporal embedding reflects the time stage of a local segment, the combination of the two can form a composite semantic representation that takes into account both the overall rhythm and the local temporal position.

[0051] Furthermore, the summation result can be input into a fully connected layer for mapping to obtain the final action temporal embedding. This mapping process can further learn the nonlinear combination relationship between action velocity priors and temporal location information, enabling segments at the same time stage under different velocity groups to have different semantic representations. For example, segments located in the middle of an operation may not have the same clinical skill significance for faster and slower videos; fully connected fusion can more effectively distinguish these differences.

[0052] For example, during a venipuncture procedure, if the mid-stage movement appears in a short video, it may indicate that the operator is skilled and decisive; while if it appears in a long video, it may indicate that the operator pauses more often and attempts repeatedly. Through the action temporal embedding generated in this step, the model can synchronously inject the action rhythm prior and temporal stage semantics into the subsequent feature modeling process.

[0053] Specifically, after obtaining the video temporal embedding, the video temporal embedding and the motion velocity text embedding corresponding to the motion velocity description group are added element-wise, and then mapped through a fully connected layer to generate the final motion temporal embedding representation. The calculation method is as follows: ; in, Indicates action timing embedding; Indicates the first Temporal embedding of a clinical skills video, This represents the semantic embedding of the action speed corresponding to its action speed group. This represents a fully connected mapping layer. The action temporal embedding is used to characterize the temporal semantic features that fuse action execution rhythm and temporal location information.

[0054] S6: Fuse the action temporal embedding with the preprocessed video feature representation to obtain the temporal feature representation; The temporal embedding of actions is fused with the preprocessed video feature representation to obtain a temporal feature representation, including: S61: Align the action timing embedding with the preprocessed video feature representation in terms of dimensions; S62: The dimension-aligned action sequence embedding and the preprocessed video feature representation are fused element by element to obtain the fused feature representation corresponding to each video segment; S63: Perform linear transformation and normalization on the fused feature representation to obtain the temporal feature representation.

[0055] Specifically, the action temporal embedding representation is added element-wise to the preprocessed video segment-level visual features to generate a temporal feature representation that integrates action speed semantic information and visual information. ; used for subsequent skill semantic decoding. Among them, This represents the preprocessed video segment-level visual features; This indicates the embedding of action sequences.

[0056] In this embodiment, after completing the construction of the action temporal embedding and the acquisition of the preprocessed video feature representation, the action temporal embedding and the preprocessed video feature representation are first aligned in dimensions. Specifically, since the action temporal embedding mainly represents the semantic information of the action speed and the temporal order information corresponding to the video segment, while the preprocessed video feature representation mainly represents the visual content features of the corresponding video segment, there may be inconsistencies in feature dimensions or different representation forms during the initial generation process. Therefore, by using unified mapping, dimension adjustment, or corresponding position matching, the action temporal embedding and the preprocessed video feature representation corresponding to each video segment can be kept consistent in feature dimensions and segment order, so as to facilitate subsequent segment-by-segment fusion. In this way, it is possible to ensure that semantic prior information and visual feature information are effectively combined in the same feature space.

[0057] After dimensional alignment, the action temporal embedding and preprocessed video feature representation are fused element-wise to obtain the fused feature representation for each video segment. Specifically, for each video segment, the corresponding action temporal embedding and the corresponding preprocessed video feature representation are superimposed element-wise at the same positions, so that the fused feature simultaneously contains the visual appearance information, action execution rhythm information, and temporal semantic information of the video segment. Through this fusion method, while preserving the visual features of the original video segment, the semantic prior introduced by the action speed description text and the stage information represented by the temporal embedding are jointly injected into the video feature representation. This enables the model not only to identify what kind of operation content appears in the video segment, but also to further identify the time stage of the operation content and the corresponding execution rhythm.

[0058] After obtaining the fused feature representation, it undergoes further linear transformation and normalization to obtain a temporal feature representation. The linear transformation reorganizes and enhances the fused multi-source features, creating a closer association between the temporal semantics of the actions and the visual content. Normalization improves the stability of the fused features in terms of numerical distribution, reducing the impact of feature fluctuations between different samples on subsequent decoding and scoring processes. The temporal feature representation obtained after this step more comprehensively characterizes the action content, execution rhythm, and operational sequence features of clinical skills videos at different time stages. It can serve as the input basis for subsequent text-guided decoding and scoring prediction for skills grading, thereby improving the accuracy and stability of automatic clinical skills scoring.

[0059] S7: Encode the preset skill level description text to obtain the skill level text embedding, and introduce a learnable memory embedding vector. Decode the temporal feature representation based on the cross-attention mechanism to obtain the decoded feature representation. Among them, reference Figure 4 As shown, the predefined skill level description text is encoded to obtain the skill level text embedding, and a learnable memory embedding vector is introduced. The temporal feature representation is then decoded based on a cross-attention mechanism to obtain the decoded feature representation, including: S71: Description text for preset skill levels The skill level description text is encoded using a pre-trained S3D text encoder to obtain the skill level text embedding. S72: Introduce learnable memory embedding vectors and construct query vectors based on skill-level text embeddings; S73: Using the query vector as the query and the temporal feature representation as the key and value, the learnable memory embedding vector is updated through a cross-attention mechanism to obtain the memory feature representation; S74: Using memory feature representation as the query and temporal feature representation as the key and value, the temporal feature representation is decoded through a cross-attention mechanism to obtain the decoded feature representation.

[0060] In this embodiment, multiple sets of skill-level description texts are first pre-defined to characterize differences in the quality levels of clinical skills. These skill-level description texts are then encoded to obtain corresponding skill-level text embeddings. The skill-level description texts can be designed based on clinical skills assessment standards, and may include descriptions of different levels such as "non-standard operation, missing key steps," "basically standardized operation, relatively complete process," and "proficient operation, coherent movements, and high quality," to depict the quality differences of clinical skills from low to high. By encoding the text, it is transformed into a vector representation with semantic expressive capabilities, enabling the model to use this semantic information to guide video features in subsequent processes.

[0061] Furthermore, a learnable memory embedding vector is introduced, and a query vector is constructed based on the skill level text embedding. The learnable memory embedding vector is used to store latent semantic features related to skill levels, and it is continuously updated during model training to adapt to the feature distribution of samples at different skill levels. By fusing the skill level text embedding with the memory embedding vector, a query vector containing skill level semantic information is constructed, enabling subsequent feature interaction processes to filter and aggregate information based on skill levels.

[0062] Specifically, the memory embedding vector Random initialization is performed, and the data is fused with skill-level text embeddings through a cross-attention mechanism. The calculation method is as follows:

[0063]

[0064] Where Q represents the query vector; K represents the key vector; and V represents the value vector; Represents the memory embedding vector; Representation of temporal characteristics; This indicates the memory decoding features of text-guided learning; This is a scaling factor used to stabilize the attention distribution.

[0065] Building upon this foundation, the learnable memory embedding vector is updated using the query vector as the query and the temporal feature representation as the key and value, through a cross-attention mechanism, resulting in a memory feature representation. In this process, key temporal information closely related to skill level discrimination can be extracted from the temporal feature representation and written into the memory embedding vector. This ensures that the updated memory feature representation not only includes skill level semantics but also incorporates dynamic operational features related to that semantics from the video, thus forming a more discriminative semantic representation.

[0066] After obtaining the memory feature representation, the memory feature representation is used as the query, and the temporal feature representation is used as the key and value. The temporal feature representation is then decoded again through a cross-attention mechanism to obtain the decoded feature representation. Through this decoding process, the key segments most relevant to skill assessment can be further filtered from the temporal features of the entire video. This makes the final decoded feature representation more focused on evaluation elements such as operational standardization, action coherence, and process integrity. This provides more discriminative and stable input features for the subsequent scoring network, effectively improving the accuracy and reliability of automatic clinical skills scoring.

[0067] S8: Input the decoded feature representation into the scoring network to obtain the predicted skill score; Among them, reference Figure 5 As shown, the decoded feature representation is input into the scoring network to obtain the predicted skill score, including: S81: Input the decoded feature representation into the scoring network, and perform feature mapping through a multi-layer fully connected network to obtain the scoring feature vector; S82: Based on the rating feature vector and mapped by the activation function, the predicted value of the skill rating is obtained.

[0068] In this embodiment, after obtaining the decoded feature representation, the decoded feature representation is input into the scoring network, and then processed layer by layer through a multi-layer fully connected network to obtain the scoring feature vector. The multi-layer fully connected network is used to extract and aggregate key information from the decoded features, so that the original high-dimensional semantic features are gradually transformed into a compact representation that is highly correlated with the skill score, thereby enhancing the ability to distinguish between samples of different clinical skill levels.

[0069] Furthermore, based on the scoring feature vector, an activation function is used to map and obtain the predicted skill score. This predicted skill score is a continuous numerical value used to characterize the operational quality level corresponding to the clinical skill video. Videos with standardized operations, complete procedures, and coherent movements receive higher scores, while videos with missing steps or lack of proficiency receive lower scores. In this way, the decoded semantic features are automatically mapped to objective and stable skill scoring results.

[0070] Specifically, for clinical skills videos The skill decoding features are input into the scoring network, and the skill decoding features are mapped through a multi-layer fully connected network to generate a scoring vector corresponding to each skill level. The calculation process for the rating vector is defined as follows: ; in, Represents the predicted score vector; Indicates the first Skill decoding features corresponding to each video; These are learnable weight parameters; For bias terms; This represents the Sigmoid activation function.

[0071] Predicted rating vector based on the output of the rating module The score error loss is calculated by comparing the corresponding real skill scores with the training of the supervised scoring module. The score error loss is defined as follows: ; in, Represents the scoring loss function; Indicates the training batch size. and They represent the first The predicted rating vector and the actual rating vector for each sample.

[0072] To enhance the discriminative power of skill decoding features in the feature space, a differential loss is introduced to constrain the skill decoding features, in which representative anchor features are selected. Positive sample characteristics and negative sample features The differential loss is calculated by optimizing the following objective. ; in, Indicates the loss of differentiation; This represents the truncation function; and These represent the distance metric functions between corresponding feature pairs. This is the interval hyperparameter.

[0073] S9: Construct a ranking relation matrix based on the actual skill score differences, and calculate the ranking graph constraint loss based on the ranking relation matrix and the predicted skill score values. Then, jointly optimize the ranking graph constraint loss and the score loss.

[0074] Among them, reference Figure 6As shown, a ranking relation matrix is ​​constructed based on the actual skill score differences, and the ranking graph constraint loss is calculated based on the ranking relation matrix and the predicted skill score values. The ranking graph constraint loss and the score loss are jointly optimized, including: S91: Construct a real ranking relationship matrix based on the differences in real skill scores of clinical skill videos within the same training batch; S92: Construct a prediction ranking matrix based on predicted skill scores; S93: Calculate the ordering graph constraint loss based on the difference between the actual ordering relation matrix and the predicted ordering relation matrix; S94: The ordering graph constraint loss and the scoring loss are weighted and fused to obtain the joint loss function; S95: Optimize parameters based on the joint loss function.

[0075] In this embodiment, a true ranking matrix is ​​first constructed based on the differences in actual skill scores among clinical skill videos within the same training batch. Specifically, each video sample in the training batch is compared pairwise, and their relative high and low relationships are determined according to the corresponding actual skill scores, thereby establishing a ranking structure among the videos. In this way, the originally independent individual score values ​​can be transformed into an overall ranking relationship, enabling the model to perceive the relative level differences between different clinical skill videos, thus providing a basis for subsequent ranking constraints.

[0076] Furthermore, based on the predicted skill scores output by the scoring network, pairwise comparisons are performed on each clinical skill video within the same training batch to construct a predicted ranking matrix. This predicted ranking matrix reflects the model's current ranking judgment of the skill levels of each video sample. By mapping the predicted ranking relationships to the actual ranking relationships, the model's learning effect at the ranking level can be evaluated, enabling the model to focus not only on the scoring error of individual samples but also on the rationality of the overall ranking.

[0077] Specifically, a true ranking graph is constructed based on the actual skill score relationships of clinical skill videos within the same training batch. In this graph, nodes represent clinical skill videos, and the true ranking relationship between node pairs is determined by the corresponding skill score differences, thus forming a true ranking relationship matrix. Simultaneously, a prediction ranking graph is constructed based on the predicted skill scores output by the model, forming a prediction ranking relationship matrix. .

[0078] Based on this, the ranking graph constraint loss is calculated according to the difference between the true ranking relation matrix and the predicted ranking relation matrix. By introducing the ranking graph constraint, the model can penalize inconsistent sample pairs during training, thereby gradually adjusting the model parameters so that the prediction results better reflect the ranking relations of real skill levels while maintaining numerical accuracy. This ranking constraint mechanism can effectively compensate for the ranking instability problem caused by training solely based on scoring errors, and improve the model's overall ability to discriminate differences in skill levels.

[0079] Specifically, the formula for calculating the ordering graph constraint loss is as follows: ; in, This represents the loss due to sorting graph constraints. This represents the number of clinical skills videos participating in the ranking constraint; i and j represent the video sample indices. Represents the weights or mask matrix elements of the sorting constraints; This represents the elements in the prediction ranking relation matrix; This represents an element in the actual sorting matrix.

[0080] Furthermore, the ranking graph constraint loss and the scoring loss are weighted and fused to obtain a joint loss function, which is then used to optimize the model parameters. The scoring loss constrains the difference between predicted and actual scores, while the ranking graph constraint loss constrains the consistency of ranking relationships between samples. Through their synergistic effect, the model simultaneously optimizes both absolute scoring accuracy and relative ranking relationships during training. Iterative optimization of the joint loss function makes the model's output skill scores more stable and exhibits good ranking consistency, thereby improving the overall performance and reliability of the automated clinical skills scoring system.

[0081] Specifically, the sorting graph loss Introduced as a ranking consistency constraint into the model training process, this is related to the score error loss generated by the scoring module. and differentiation loss The total loss function that constitutes the model is: ; in, and These represent the weight coefficients of the ranking graph loss term and the differentiation loss term, respectively, used to balance the contribution of each loss term during model training.

[0082] By minimizing the total loss function mentioned above, while regressing clinical skill scores, a ranking graph constraint is introduced to maintain the consistency of skill ranking relationships between videos, and differential loss is used to enhance the discriminability of different skill samples in the feature space, thereby achieving joint optimization of model parameters.

[0083] This application also provides a text-guided intelligent assessment system for clinical skills based on ordination graph constraints, the system comprising: The first module is used to perform frame sampling and segmentation processing on the input clinical skills video, and extract segment-level visual feature sequences; The second module is used to map and normalize the segment-level visual feature sequences to obtain preprocessed video feature representations. The third module is used to sort and group videos based on video duration and encode preset motion speed description text to obtain motion speed text embedding. The fourth module is used to generate video temporal embeddings based on the temporal order of video segments; The fifth module is used to construct action temporal embeddings based on video temporal embeddings and action speed text embeddings; The sixth module is used to fuse the action temporal embedding with the preprocessed video feature representation to obtain the temporal feature representation; The seventh module is used to encode the preset skill level description text to obtain the skill level text embedding, and introduce a learnable memory embedding vector. Based on the cross attention mechanism, the temporal feature representation is decoded to obtain the decoded feature representation. The eighth module is used to input the decoded feature representation into the scoring network to obtain the predicted skill score; The ninth module is used to construct a ranking relation matrix based on the actual skill score differences, calculate the ranking graph constraint loss based on the ranking relation matrix and the predicted skill score values, and jointly optimize the ranking graph constraint loss and the score loss.

[0084] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0085] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0086] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0087] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0088] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0089] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0090] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0092] This application provides a text-guided intelligent evaluation method for clinical skills based on ordination graph constraints. This method extracts stable and discriminative segment-level visual features from clinical skills videos through frame sampling, segmentation, and feature mapping. These features are then grouped and ranked according to video duration. Joint modeling is performed using text embeddings describing action speed and video temporal embeddings to effectively characterize the rhythm and temporal evolution of actions during clinical procedures, improving the ability to recognize operational standardization and coherence. Furthermore, by fusing action temporal embeddings with video feature representations and combining skill-level text embeddings, learnable memory embedding vectors, and cross-attention mechanisms for decoding, deep alignment between video process information and scoring semantics is achieved. This enables precise discrimination of key operational segments for different skill levels, thereby improving the accuracy and interpretability of the scoring results. Simultaneously, an ordination relationship matrix based on real scoring differences is constructed, and ordination graph constraint loss and scoring loss are jointly optimized. This ensures scoring regression accuracy while strengthening the relative ordination relationship between samples, effectively avoiding inconsistencies in scoring results and improving the overall stability and consistency of the evaluation. Ultimately, this achieves a more objective, refined, and reliable intelligent evaluation of clinical skill levels.

[0093] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0094] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0095] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0096] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0097] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A text-guided intelligent assessment method for clinical skills based on ordination graph constraints, characterized in that, The method includes the following steps: The input clinical skills video is subjected to frame sampling and segmentation processing to extract segment-level visual feature sequences; The segment-level visual feature sequence is mapped and normalized to obtain a preprocessed video feature representation; The videos are sorted and grouped based on their duration, and the preset motion speed description text is encoded to obtain the motion speed text embedding. Generate video temporal embeddings based on the temporal order of video segments; An action-time embedding is constructed based on the video temporal embedding and the action speed text embedding; The action timing embedding is fused with the preprocessed video feature representation to obtain a timing feature representation; The preset skill level description text is encoded to obtain the skill level text embedding, and a learnable memory embedding vector is introduced. The temporal feature representation is decoded based on the cross-attention mechanism to obtain the decoded feature representation. The decoded feature representation is input into the scoring network to obtain the predicted skill score; A ranking relation matrix is ​​constructed based on the actual skill score differences, and a ranking graph constraint loss is calculated based on the ranking relation matrix and the predicted skill score values. The ranking graph constraint loss and the score loss are then jointly optimized.

2. The method according to claim 1, characterized in that, The process of frame sampling and segmenting the input clinical skills video to extract segment-level visual feature sequences includes: The input clinical skills video is uniformly sampled according to a preset number of sampling frames to obtain a video frame sequence; The video frame sequence is segmented based on a preset number of frames per segment. When the number of frames is insufficient, frame interpolation is performed to fill in the gaps, resulting in multiple video segments. Each video segment is input into a dilated 3D convolutional network for encoding to obtain the corresponding segment-level visual features; The segment-level visual features are combined in chronological order to obtain a segment-level visual feature sequence.

3. The method according to claim 1, characterized in that, The process of sorting and grouping videos based on video duration and encoding preset motion speed description text to obtain motion speed text embedding includes: Multiple sets of action speed description texts are preset to characterize the differences in the execution speed of clinical skills, and the action speed description texts are encoded to obtain the corresponding action speed text embeddings; The clinical skills videos are sorted based on their duration and divided into multiple action speed groups according to preset grouping rules. Based on the multiple action speed groups, corresponding action speed text embeddings are assigned to each clinical skill video, and the action speed text embeddings are used as semantic prior information of action speed.

4. The method according to claim 1, characterized in that, The step of generating video temporal embedding based on the temporal order of video segments includes: Based on the temporal order of video segments in the clinical skills videos, determine the time index position corresponding to each video segment; Based on the time index position, a corresponding temporal embedding representation is generated for each video segment; The temporal embedding representation is combined with a preset time position encoding function to obtain video temporal embedding.

5. The method according to claim 4, characterized in that, The formula for the time position encoding function is as follows: ； in, Indicates the first Embed the time location corresponding to each clinical skills video; Indicates an embedded dimension index, when When the number is even, the sine function is used. When the number is odd, the cosine function is used; Indicates the feature embedding dimension; Indicates the first Temporal location-related features of each clinical skills video; Indicates the first The overall time rhythm or time scale parameter of a clinical skills video.

6. The method according to claim 1, characterized in that, The step of fusing the action timing embedding with the preprocessed video feature representation to obtain a timing feature representation includes: The action timing embedding is dimensionally aligned with the preprocessed video feature representation; The dimension-aligned action temporal embedding and the preprocessed video feature representation are fused element-wise to obtain the fused feature representation corresponding to each video segment; The fused feature representation is subjected to linear transformation and normalization to obtain the temporal feature representation.

7. The method according to claim 1, characterized in that, The process involves encoding the preset skill level description text to obtain a skill level text embedding, introducing a learnable and memory-based embedding vector, and decoding the temporal feature representation based on a cross-attention mechanism to obtain a decoded feature representation, including: The preset skill level description text is encoded to obtain the skill level text embedding; A learnable memory embedding vector is introduced, and a query vector is constructed based on the skill-level text embedding. Using the query vector as the query and the temporal feature representation as the key and value, the learnable memory embedding vector is updated through a cross-attention mechanism to obtain the memory feature representation; Using the memory feature representation as the query and the temporal feature representation as the key and value, the temporal feature representation is decoded through a cross-attention mechanism to obtain the decoded feature representation.

8. The method according to claim 1, characterized in that, The step of inputting the decoded feature representation into the scoring network to obtain the skill score prediction value includes: The decoded feature representation is input into the scoring network, and feature mapping is performed through a multi-layer fully connected network to obtain the scoring feature vector; Based on the rating feature vector, the skill rating prediction value is obtained by mapping through an activation function.

9. The method according to claim 1, characterized in that, The process involves constructing a ranking relationship matrix based on the actual skill score differences, calculating the ranking graph constraint loss based on the ranking relationship matrix and the predicted skill scores, and jointly optimizing the ranking graph constraint loss and the score loss, including: Construct a real ranking relationship matrix based on the differences in real skill scores of clinical skill videos within the same training batch; Construct a prediction ranking matrix based on the predicted skill scores; The ranking graph constraint loss is calculated based on the difference between the actual ranking relation matrix and the predicted ranking relation matrix; The sorting graph constraint loss and the scoring loss are weighted and fused to obtain the joint loss function; The parameters are optimized based on the joint loss function.

10. The method according to claim 9, characterized in that, The formula for calculating the ordering graph constraint loss is as follows: ； in, This represents the loss due to sorting graph constraints. This represents the number of clinical skills videos participating in the ranking constraint; i and j represent the video sample indices. Represents the weights or mask matrix elements of the sorting constraints; This represents the elements in the prediction ranking relation matrix; This represents an element in the actual sorting matrix.

Images