Desktop operation track prediction and planning scheduling method based on multi-modal large model

Abstract

The invention relates to the technical field of multi-modal processing, and discloses a desktop operation track prediction and planning scheduling method based on a multi-modal large model, and the method comprises the following steps: constructing the multi-modal large model for fine adjustment in the medical field, and analyzing the multi-modal information in the desktop operation of a doctor; based on the multi-modal information analyzed by the multi-modal large model, performing multi-dimensional medical context sensing processing, and generating a structured medical context feature vector; according to the method, deep fusion analysis of multi-modal information in desktop operation of a doctor is realized by constructing a fine-tuning multi-modal large model in the medical field, and the operation intention of the doctor can be analyzed based on a multi-modal input combination of a screen interface image, a voice instruction and a keyboard and mouse operation track, so that analysis errors caused by information source conflicts are avoided, and the analysis accuracy is improved. And through a collaborative analysis mechanism of a multi-modal large model, the consistency and accuracy of operation intention recognition are ensured, and a reliable basis is provided for subsequent trajectory prediction.

Description

Technical Field

[0001] This invention relates to the field of multimodal processing technology, specifically to a method for predicting and scheduling desktop operation trajectories based on a large multimodal model. Background Technology

[0002] In recent years, with the development of mobile Internet technology and the upgrading of electronic device hardware configurations, there are more and more types of applications with increasingly richer functions. With the development of computer terminals and electronic product terminals, operating systems have become an indispensable component of various terminal products.

[0003] Currently, in desktop operation assistance systems in the medical field, because doctors' diagnosis and treatment tasks involve multiple heterogeneous information sources, existing operation prediction methods cannot integrate multimodal information such as screen interface, voice commands, and keyboard and mouse trajectories when parsing doctors' desktop operations. When there are semantic conflicts and temporal deviations between multimodal information, it will lead to errors in parsing operation intentions and affect the accuracy of subsequent trajectory prediction.

[0004] Therefore, a desktop operation trajectory prediction and planning scheduling method based on a multimodal large model is proposed to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a desktop operation trajectory prediction and planning scheduling method based on a multimodal large model, which solves the problem mentioned in the background technology of operation intention parsing errors affecting the accuracy of subsequent trajectory prediction.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a desktop operation trajectory prediction and planning scheduling method based on a multimodal large model, the method comprising the following steps: Construct a fine-tuned multimodal large model in the medical field to analyze multimodal information in doctors' desktop operations; Based on the multimodal information parsed from the multimodal large model, multidimensional medical context perception processing is performed to generate structured medical context feature vectors. Based on the aforementioned multimodal large model and medical context feature vectors, doctor desktop operation trajectory prediction processing is performed to generate operation trajectory prediction data; Based on the operation trajectory prediction data and medical context feature vector, the diagnosis and treatment task planning and scheduling process is performed to generate task planning and scheduling data. Based on the task planning and scheduling data and real-time feedback from doctors, dynamic optimization and feedback closed-loop processing are performed to adjust the prediction and scheduling results in real time and update the parameters of the multimodal large model.

[0007] Preferably, the construction of a finely tuned multimodal large model in the medical field includes the following steps: Construct a multimodal medical dataset, including desktop operation multimodal logs, structured medical knowledge, and departmental personalized operation data. The desktop operation multimodal logs include screen screenshots, doctor's voice command recordings, text input records, and keyboard and mouse operation trajectories. We employ instruction fine-tuning and contrastive learning strategies to fine-tune a general multimodal large model for medical scenarios, with the goal of parsing multimodal inputs into operational intents and incorporating medical rule constraints. The finely tuned multimodal large model is lightweighted using model distillation technology to obtain a lightweight model suitable for local deployment and real-time interaction on doctors' office terminals.

[0008] Preferably, the fine-tuning of the general multimodal large model for medical scenarios specifically includes: The training task is to map multimodal input combinations to operation intent parsing. When the input screen image displays "Unsaved medical record pop-up" and the voice command "Save medical record first", the output intent is "Execute medical record save operation". At the same time, medical rule constraints are embedded. When the input text "Penicillin allergy history" and the interface image "Amoxicillin prescription" are displayed, the contraindication warning intent is triggered. During the fine-tuning process, the cross-entropy loss function is used to optimize the model parameters. The model parameters are iteratively adjusted by calculating the cross-entropy loss value between the predicted intent distribution and the true intent distribution.

[0009] Preferably, the multi-dimensional medical context-aware processing includes the following steps: Patient condition context processing: Connecting to the hospital's HIS and EMR systems to extract basic patient information, diagnostic and treatment data, and the urgency of the condition. Basic information includes age, gender, and weight; diagnostic and treatment data includes diagnostic results, laboratory indicators, allergy history, and previous medications; and the urgency of the condition is divided into outpatient routine and emergency rescue levels. Treatment process context layer processing: Determine the treatment stage based on the current operation progress, including consultation, diagnosis, prescription and archiving, and associate it with the corresponding medical rules; Personalized contextual processing for doctors: Analyze historical operation logs to extract doctors' operational habits, including time preferences, template usage habits, and shortcut key frequency; The three layers of contextual information are fused into a 512-dimensional feature vector by knowledge graph embedding technology. Specifically, this is achieved by concatenating the original contextual data of patients, diagnosis and treatment and doctors, and then vectorizing them using an embedding layer based on the TransE knowledge graph.

[0010] Preferably, the personalized context layer processing for doctors specifically includes the following steps: Collect doctors' historical operation log data; The historical operation logs are analyzed based on clustering algorithms to extract personalized features that characterize doctors' operating habits. These personalized features include time distribution features and operation preference features. The extracted personalized features are then quantitatively characterized. The personalized features occupy a preset weight ratio in the subsequent planning and scheduling, and can be dynamically adjusted according to the doctor's actual operation feedback.

[0011] Preferably, the process of predicting the doctor's desktop operation trajectory includes the following steps: Multimodal information parsing: The multimodal large model receives screen interface images, real-time doctor operations, and medical context feature vectors, and parses and outputs the current operation intent; Single-step action prediction: Combining the current operational intent and medical context, output the next atomic operation and its confidence level; Continuous operation trajectory planning: Based on the current operation intention and the predicted single-step action, the plan performs rolling optimization planning of future multi-step operation trajectories, and incorporates medical rules for verification during the planning process.

[0012] Preferably, the rolling time-domain optimization planning specifically includes: Starting from the current moment, predict the trajectory for the next k steps, and re-optimize the subsequent trajectory after each step. During the optimization process, a dynamic programming algorithm is used to minimize the trajectory deviation, and the cost function is calculated based on the state and operation actions; Medical rule verification is achieved through a rule engine, which automatically corrects the missing key steps in the trajectory.

[0013] Preferably, the process of planning and scheduling medical treatment tasks includes the following steps: Task extraction and modeling: Extract tasks to be processed from the doctor's desktop window and to-do reminders, and label task attribute information, including urgency, complexity and associated patient type; Priority ranking: Based on task attribute information, a multi-factor weighted scoring method is used to calculate the priority score of each task; Time planning and scheduling: Based on task priority scoring and combined with the task time predicted by the operation trajectory and the doctor's work intervals, a visual schedule that supports real-time adjustment is generated.

[0014] Preferably, the calculation of the priority score for each task using a multi-factor weighted scoring method includes the following specific steps: Scoring factor definition and weight configuration: Establish a scoring system consisting of three core factors: task urgency, task complexity, and doctor's operational habits and preferences, and assign dynamically adjustable initial weights to each factor. Urgency rating mapping: Based on a predefined medical rule base, the urgency level in task attributes is mapped to a standardized urgency rating value, with emergency tasks automatically receiving the highest rating. Complexity score calculation: Analyze the clinical data characteristics related to the task, and calculate the complexity score based on the number of diagnostic and treatment steps, the number of complications involved, and the level of medical resources required. Habit preference score extraction: Quantitatively extract operational habit preference scores from doctors' personalized contextual features, including weighted statistics on historical completion efficiency and common template dependence for specific task types; Dynamic weighted fusion: The weight ratios of the three factors are dynamically adjusted according to the needs of real-time diagnosis and treatment scenarios. A linear weighted summation algorithm is used to generate the final comprehensive task priority score. The calculation formula is as follows: ;in For comprehensive priority scoring, Assigning weights based on urgency To score the urgency level, For complexity weights, Score for complexity. For habitual weighting, Score based on habit; The weighting factors satisfy the normalization constraint. ; Scoring result verification and correction: The calculated comprehensive score is compared with the historical priority threshold. If a scoring logic conflict is detected, the rule engine is triggered to make an automatic correction.

[0015] Preferably, the dynamic optimization and feedback closed-loop processing includes the following steps: Real-time feedback and adjustment: Based on the execution effect of task planning and scheduling data, doctors can correct the predicted trajectory and scheduling results through voice commands and interface buttons. The corrected data is sent back to the multimodal large model in real time for incremental training, generating updated model parameters and feedback logs. Periodic optimization: Based on the updated model parameters and feedback logs, the prediction accuracy and scheduling fit rate are extracted from the multimodal large model every week, and the model weights are optimized using the gradient descent algorithm to minimize the loss function, thus completing the periodic adjustment of the model parameters.

[0016] Compared with existing technologies, this invention provides a desktop operation trajectory prediction and planning scheduling method based on a multimodal large model, which has the following beneficial effects: 1. In this invention, by constructing a finely tuned multimodal large model in the medical field, a deep fusion and analysis of multimodal information in doctors' desktop operations is achieved. Based on the multimodal input combination of screen interface images, voice commands, and keyboard and mouse operation trajectories, the doctor's operation intention can be analyzed, avoiding analysis errors caused by conflicting information sources. Through the collaborative analysis mechanism of the multimodal large model, the consistency and accuracy of operation intention recognition are guaranteed, providing a reliable foundation for subsequent trajectory prediction.

[0017] 2. In this invention, a deep association mechanism between patient condition, diagnosis and treatment process and doctor's personalized habits is established through multi-dimensional medical context perception processing. Based on structured medical context feature vectors, the urgency, complexity and personalized needs of diagnosis and treatment tasks are dynamically evaluated to achieve quantitative ranking of task priorities. Through medical context-driven scheduling decisions, the timeliness of diagnosis and treatment task processing and the rationality of resource allocation are improved.

[0018] 3. In this invention, a compliance assurance system that runs through the entire process of trajectory generation is constructed through the deep integration of medical rule verification mechanism and operation trajectory prediction. Based on real-time matching and verification of medical rule base, it actively identifies and avoids medical risks such as drug contraindications and violation of operation procedures. Through the closed-loop linkage of trajectory generation and rule verification, the clinical safety and reliability of the final output operation trajectory are guaranteed, providing core assurance for medical quality and safety. Attached Figure Description

[0020] Figure 1 This is a flowchart of the desktop operation trajectory prediction and planning scheduling method based on a multimodal large model according to the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] For specific implementation examples, please refer to: Figure 1 A desktop operation trajectory prediction and planning scheduling method based on a multimodal large model, the method includes the following steps: Construct a fine-tuned multimodal large model in the medical field to analyze multimodal information in doctors' desktop operations; Based on the multimodal information parsed by the multimodal large model, multi-dimensional medical context perception processing is performed to generate structured medical context feature vectors. Based on a multimodal large model and medical context feature vectors, the doctor's desktop operation trajectory is predicted to generate operation trajectory prediction data. Based on operation trajectory prediction data and medical context feature vectors, the diagnosis and treatment task planning and scheduling process is performed to generate task planning and scheduling data. Based on task planning and scheduling data and real-time feedback from doctors, dynamic optimization and feedback loop processing are performed to adjust prediction and scheduling results in real time and update the parameters of the multimodal large model.

[0024] Building a finely tuned multimodal large model for the medical field includes the following steps: Construct a multimodal medical dataset, including desktop operation multimodal logs, structured medical knowledge, and departmental personalized operation data. The desktop operation multimodal logs include screen screenshots, doctor's voice command recordings, text input records, and keyboard and mouse operation trajectories. We employ instruction fine-tuning and contrastive learning strategies to fine-tune a general multimodal large model for medical scenarios, with the goal of parsing multimodal inputs into operational intents and incorporating medical rule constraints. The fine-tuned multimodal large model is lightweighted using model distillation technology to obtain a lightweight model suitable for local deployment and real-time interaction on doctors' office terminals. The process includes the following steps: Teacher model inference: The fine-tuned multimodal large model is used as the teacher model to infer the samples in the multimodal medical dataset and obtain the prediction results output by the model. Student Model Training: Construct a student model with a smaller parameter size, using the prediction results of the teacher model as the training objective. Train the student model using the cross-entropy loss function and gradient descent algorithm. The distillation loss function is calculated as follows: ;in This represents the distillation loss value. The number of training samples, For the teacher model to the first The predicted probability of a sample. For the student model to the first The predicted probability of a sample. For indexing; Model performance evaluation: The prediction accuracy of the lightweight student model is evaluated on an independent validation set. The accuracy is calculated using the following formula: ; in To predict the correct number of samples, To verify the total number of samples; Local deployment and integration: Deploy the trained lightweight model to the doctor's office terminal and achieve real-time interactive interface with the existing operating system; Fine-tuning of a general multimodal large model for medical scenarios specifically includes: The training task is to map multimodal input combinations to operation intent parsing. When the input screen image displays "Unsaved medical record pop-up" and the voice command "Save medical record first", the output intent is "Execute medical record save operation". At the same time, medical rule constraints are embedded. When the input text "Penicillin allergy history" and the interface image "Amoxicillin prescription" are displayed, the contraindication warning intent is triggered. The fine-tuning process uses the cross-entropy loss function to optimize model parameters. This involves iteratively adjusting the model parameters by calculating the cross-entropy loss between the predicted intent distribution and the true intent distribution, and includes the following steps: Acquire training batch data, including multimodal input samples and their corresponding real operation intent labels; By inputting multimodal input samples into a large multimodal model, the probability distribution of the intention predicted by the model is obtained. The cross-entropy loss between the predicted intent probability distribution and the true intent label is calculated using the following formula: ;in This represents the cross-entropy loss value. The number of training samples, For the first The true label of each sample For the model to the first The predicted probability distribution of each sample. For indexing; Based on the cross-entropy loss value, the gradient of the model parameters is calculated using the backpropagation algorithm; An optimizer is used to update the parameters of a large multimodal model based on gradients in order to minimize the cross-entropy loss. Multi-dimensional medical context-aware processing includes the following steps: Patient condition context processing: Connecting to the hospital's HIS and EMR systems to extract basic patient information, diagnostic and treatment data, and the urgency of the condition. Basic information includes age, gender, and weight; diagnostic and treatment data includes diagnostic results, laboratory indicators, allergy history, and previous medications; and the urgency of the condition is divided into outpatient routine and emergency rescue levels. Treatment process context layer processing: Determine the treatment stage based on the current operation progress, including consultation, diagnosis, prescription and archiving, and associate it with the corresponding medical rules; Personalized contextual processing for doctors: Analyze historical operation logs to extract doctors' operational habits, including time preferences, template usage habits, and shortcut key frequency; The three layers of contextual information are fused into a 512-dimensional feature vector using knowledge graph embedding technology. Specifically, this is achieved by concatenating the original contextual data of the patient, diagnosis, and doctor, and then vectorizing it using an embedding layer based on the TransE knowledge graph. The steps include: Construct a medical knowledge graph, where entities include diseases, drugs, and diagnostic procedures, and relationships include "belongs to", "contraindicated", and "treatment"; The TransE model is used to learn vector representations of entities and relations in knowledge graphs. Its optimization objective is to minimize the difference between the sum of the head entity vector, relation vector and tail entity vector. For entities in the context of the patient's condition and the context of the diagnosis and treatment process, query their corresponding vector representations in the TransE model; The entity vector obtained from the query is concatenated with the doctor's personalized contextual feature vector; The concatenated high-dimensional vector is then subjected to dimensionality reduction processing to generate a 512-dimensional feature vector.

[0025] Personalized contextual processing by doctors specifically includes the following steps: Collect doctors' historical operation log data; Historical operation logs are analyzed using clustering algorithms to extract personalized features representing doctors' operating habits. These personalized features include time distribution characteristics and operation preference characteristics. The analysis of historical operation logs based on clustering algorithms includes the following steps: The historical operation logs are preprocessed to extract operation type, operation timestamp, and operation object identifier as features; The K-means clustering algorithm is used to perform cluster analysis on the preprocessed operational feature vectors, and the optimal number of clusters is determined based on the silhouette coefficient. ; The operational feature pattern corresponding to each cluster center point is defined as a doctor's operational habit pattern. Calculate the current doctor's operation feature vector Clustering centers of various habitual patterns The distance is calculated using the Euclidean distance formula as follows: ;in For feature dimension, Let be the Euclidean distance, and let be the distance between the current vector and the _th vector, respectively. The cluster centers at the in Values ​​on each feature For indexing; The extracted personalized features are quantitatively characterized; Personalized features occupy a certain preset weight in subsequent planning and scheduling, and can be dynamically adjusted based on feedback from doctors' actual operations.

[0026] The steps involved in predicting and processing doctors' desktop operation trajectories are as follows: Multimodal information parsing: The multimodal large model receives screen interface images, real-time doctor operations, and medical context feature vectors, and parses and outputs the current operation intent; Single-step action prediction: Combining the current operational intent and medical context, output the next atomic operation and its confidence level; Continuous operation trajectory planning: Based on the current operation intention and the predicted single-step action, the rolling optimization planning of future multi-step operation trajectories is carried out, and medical rules are incorporated into the planning process for verification. Continuous operation trajectory planning includes the following steps: Trajectory initialization: Based on the current operational intent and the predicted next single-step action, initialize the future... Candidate operation trajectory sequence for each step The preset trajectory length; Rolling optimization iteration: Starting from the currently initialized operation trajectory, rolling temporal optimization is performed. In each iteration, a multimodal large model is used to predict the possible subsequent state after executing a candidate action at the current position, and the rationality score of the action is evaluated based on medical context features. The formula for calculating the rationality score is as follows: ;in For the rationality score function, Indicates the current desktop operation status. Indicates the candidate operation action. Represents a medical context feature vector. This represents the inference and computation process of a multimodal large model. Indicates the state Next action Reasonableness score; The action that yields the highest cumulative rationality is selected and added to the trajectory sequence. The cumulative rationality calculation formula is as follows: ;in To accumulate reasonableness scores, Indicates candidate trajectories, As a discount factor, The preset trajectory length, For time step index, In time step state, In time step The action; Online medical rule verification: After each step of trajectory planning, the medical rule verification module is called in real time to match the currently planned partial trajectory with the patient's context information. If a taboo rule is triggered, the process will backtrack to the previous decision point and exclude candidate actions that would lead to violations. Trajectory evaluation and output: When the planned number of steps reaches the preset length and the termination condition is met, the continuous operation trajectory with the highest current cumulative rationality and that passes the medical rule verification is output as the final prediction result; Incorporating medical regulations into the planning process for verification includes the following steps: Construct a medical rule base, which includes rules on drug incompatibilities, constraints on the order of diagnosis and treatment procedures, and rules on examinations for specific contraindications of patients. After each step of the operation trajectory is predicted, the predicted operation action is matched and verified against the current patient context information in the medical rule base. When taboo and constraint rules in the rule base are triggered, a warning message is generated and the generation of the operation trajectory is prohibited. Record the verification results as a basis for evaluating the reliability of the trajectory.

[0027] Rolling time-domain optimization planning specifically includes: Starting from the current moment, predict the trajectory for the next k steps, and re-optimize the subsequent trajectory after each step. The optimization process employs a dynamic programming algorithm to minimize trajectory deviation, calculated based on a cost function derived from the state and operational actions. This includes the following steps: The optimization process employs a dynamic programming algorithm to minimize trajectory deviation, calculated based on a cost function derived from the state and operational actions. This includes the following steps: Define a state space where each state is a combination of track geometry parameters at a specific mileage position, denoted as state. ; Define the decision variable as the orbital odometer calibration action in the current state, denoted as decision. ; Constructing a value function , indicating from state The minimum cumulative trajectory deviation to the target state; Starting from the target state, it iterates backwards, calculating the optimal value function and corresponding optimal decision for each state. The core update formula is: ;in Indicates from state The minimum cumulative trajectory deviation to the target state. Indicates the current state The following are optional calibration decisions. Indicates the state Make an executive decision The resulting instantaneous trajectory deviation cost, Represents the state transition function. Indicates minimization; Medical rule verification is achieved through a rule engine, which automatically corrects the missing key steps in the trajectory.

[0028] The process of planning and scheduling medical treatment tasks includes the following steps: Task extraction and modeling: Extract tasks to be processed from the doctor's desktop window and to-do reminders, and label task attribute information, including urgency, complexity and associated patient type; Priority ranking: Based on task attribute information, a multi-factor weighted scoring method is used to calculate the priority score of each task; Time planning and scheduling: Based on task priority scoring and combined with the task time predicted by the operation trajectory and the doctor's work intervals, a visual schedule that supports real-time adjustment is generated.

[0029] The calculation of the priority score for each task using a multi-factor weighted scoring method includes the following specific steps: Scoring factor definition and weight configuration: Establish a scoring system consisting of three core factors: task urgency, task complexity, and doctor's operational habits and preferences, and assign dynamically adjustable initial weights to each factor. Urgency rating mapping: Based on a predefined medical rule base, the urgency level in task attributes is mapped to a standardized urgency rating value, with emergency tasks automatically receiving the highest rating. Complexity score calculation: Analyze the clinical data characteristics related to the task, and calculate the complexity score based on the number of diagnostic and treatment steps, the number of complications involved, and the level of medical resources required. Habit preference score extraction: Quantitatively extract operational habit preference scores from doctors' personalized contextual features, including weighted statistics on historical completion efficiency and common template dependence for specific task types; Dynamic weighted fusion: The weight ratios of the three factors are dynamically adjusted according to the needs of real-time diagnosis and treatment scenarios. A linear weighted summation algorithm is used to generate the final comprehensive task priority score. The calculation formula is as follows: ;in For comprehensive priority scoring, Assigning weights based on urgency To score the urgency level, For complexity weights, Score for complexity. For habitual weighting, Score based on habit; The weighting factors satisfy the normalization constraint. Scoring result verification and correction: The calculated comprehensive score is compared with the historical priority threshold. When a scoring logic conflict is detected, the rule engine is triggered to automatically correct it so that the scoring result conforms to the clinical diagnosis and treatment process.

[0030] Dynamic optimization and feedback loop processing include the following steps: Real-time feedback and adjustment: Based on the execution effect of task planning and scheduling data, doctors can correct the predicted trajectory and scheduling results through voice commands and interface buttons. The corrected data is sent back to the multimodal large model in real time for incremental training, generating updated model parameters and feedback logs. Periodic optimization: Based on the updated model parameters and feedback logs, prediction accuracy and scheduling fit are extracted from the multimodal large model weekly, and the model weights are optimized using the gradient descent algorithm to minimize the loss function, thus completing the periodic adjustment of model parameters, including the following steps: Sample a batch of training data from the feedback log; Input the batch data into the updated multimodal large model and calculate the loss function value under the current model parameters; Calculate the gradient of the loss function with respect to the model weights; Update the model weights according to the gradient and the preset learning rate using the following formula: ;in For the updated model weights, The model weights before the update. For learning rate, loss function Gradient with respect to the weights; Repeat the above steps until the preset number of iterations is reached and the loss function converges.

[0031] The operation steps of the desktop operation trajectory prediction and planning scheduling method based on a multimodal large model are as follows: Step 1: Multimodal Information Fusion and Intent Parsing First, a multimodal large-scale model finely tuned in the medical field is used to comprehensively perceive the doctor's desktop operating environment. This method collects multimodal data in real time, including screen interface images, voice command recordings, and keyboard and mouse operation trajectories. Information fusion is achieved through a cross-modal attention mechanism. The multimodal large-scale model performs collaborative analysis of input signals based on pre-trained medical knowledge, mapping scattered visual, voice, and behavioral features into a unified representation of operating intent. This process achieves deep information fusion through the cross-modal attention mechanism of the multimodal large-scale model, eliminating information conflicts between modalities, thereby forming an accurate understanding of the doctor's current operating goal.

[0032] Step 2: Medical Context Awareness and Feature Construction This method synchronously connects to the hospital information system to dynamically acquire three types of medical context: patient condition data, treatment process status, and doctor's work habits. It uses knowledge graph embedding technology to structure discrete medical information and construct a context representation system containing 512-dimensional feature vectors. This system can accurately reflect the urgency, operational complexity, and personalized needs of the treatment scenario, providing comprehensive context-aware support for subsequent decision-making. The feature vectors are spatially mapped through the TransE model to ensure that semantically related medical concepts maintain reasonable geometric relationships in the vector space.

[0033] Step 3: Operation Trajectory Prediction and Rule Validation Based on the aforementioned operational intent and medical context features, this method uses rolling temporal optimization planning to generate an operational trajectory prediction sequence. It searches for the optimal path in the operational state space using a dynamic programming algorithm, predicts multiple future operational actions each time, and calculates the rationality score of each action. During the trajectory generation process, it synchronously calls the medical rule base for real-time verification and performs compliance checks on each predicted action. When drug contraindications and process violations are detected, a backtracking mechanism is immediately triggered to replan a safe and reliable operational path.

[0034] Step 4: Intelligent Task Scheduling and Resource Optimization This method performs intelligent scheduling based on the predicted operation trajectory and the characteristic attributes of the tasks to be processed. A multi-factor weighted scoring method is used to quantitatively evaluate task priority, comprehensively considering task urgency, complexity, and doctor preference factors. A comprehensive score is generated through linear weighted fusion, and a visual scheduling scheme is constructed accordingly. This method dynamically monitors resource status and scenario changes, automatically adjusting the scheduling order when emergency tasks and resource conflicts occur, ensuring that high-priority tasks are processed promptly.

[0035] Step 5: Dynamic Feedback and Model Optimization The method establishes a complete closed-loop optimization mechanism, continuously improving the prediction and scheduling results through real-time feedback from doctors. It uses the gradient descent algorithm to optimize model parameters based on operational accuracy and scheduling fit rate indicators, realizing incremental learning of a multimodal large model. The model weights and rule base content are updated weekly, enabling the method to adapt to changes in the diagnosis and treatment process and the evolution of doctors' operating habits, and maintaining continuous improvement in prediction accuracy and scheduling effectiveness.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0037] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for desktop operation trajectory prediction and scheduling based on a multi-modal large model, characterized in that: The method comprises the following steps: Building a fine-tuned multi-modal large model in the medical field, analyzing multi-modal information in the doctor's desktop operation; Based on the multi-modal information analyzed by the multi-modal large model, multi-dimensional medical context perception processing is carried out to generate a structured medical context feature vector; Based on the multi-modal large model and the medical context feature vector, doctor desktop operation trajectory prediction processing is carried out to generate operation trajectory prediction data; Based on the operation trajectory prediction data and the medical context feature vector, diagnosis and treatment task planning and scheduling processing is carried out to generate task planning and scheduling data; Based on the task planning and scheduling data and the doctor's real-time feedback information, dynamic optimization and feedback closed loop processing is carried out to adjust the prediction and scheduling results in real time, and update the multi-modal large model parameters.

2. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 1, characterized in that: The method comprises the following steps: Building a multi-modal medical data set, including desktop operation multi-modal logs, structured medical knowledge and department personalized operation data, wherein the desktop operation multi-modal logs cover screen interface screenshots, doctor voice instruction recordings, text input records and keyboard and mouse operation trajectories; Using instruction fine-tuning and contrast learning strategy to fine-tune the general multi-modal large model in the medical scene, taking multi-modal input to operation intention analysis as the task goal, and integrating medical rule constraints; Through model distillation technology, the fine-tuned multi-modal large model is lightened to obtain a lightweight model suitable for local deployment and real-time interaction of the doctor's office terminal.

3. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 2, characterized in that: The general multi-modal large model fine-tuned in the medical scene specifically includes: Mapping the multi-modal input combination to the operation intention analysis as the training task, when the input screen image displays "unsaved medical record pop-up window" and the voice instruction "save the medical record first", the output intention is "execute the medical record saving operation", and at the same time, the medical rule constraints are embedded, when the input text is "penicillin allergy history" and the interface image is "amoxicillin prescription", the contraindication warning intention is triggered; In the fine-tuning process, the cross-entropy loss function is used to optimize the model parameters, and the cross-entropy loss value between the predicted intention distribution and the real intention distribution is calculated to iteratively adjust the model parameters.

4. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 1, characterized in that: The multi-dimensional medical context perception processing comprises the following steps: Patient condition context layer processing: interface with hospital HIS and EMR systems to extract patient basic information, diagnosis and treatment data, and disease severity, basic information includes age, gender and weight, diagnosis and treatment data includes diagnosis results, test indicators, allergy history and past medication, and disease severity is divided into outpatient routine, emergency rescue and other levels; Diagnosis and treatment process context layer processing: determine the diagnosis and treatment stage based on the current operation progress, including inquiry, diagnosis, prescription and archiving, and associate the corresponding medical rules; Doctor personalized context layer processing: analyze historical operation logs to extract doctor operation habit features, including time preference, template usage habit and shortcut key frequency; Through knowledge graph embedding technology, the three-layer context information is fused into a 512-dimensional feature vector, specifically by concatenating patient, diagnosis and treatment, and doctor context original data, and applying the embedding layer based on TransE knowledge graph for vectorization.

5. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 4, characterized in that: The doctor personalized context layer processing specifically comprises the following steps: Collecting doctor historical operation log data; Analyzing the historical operation log based on a clustering algorithm to extract personalized features representing the doctor's operation habits, including time distribution features and operation preference features; Quantitatively representing the extracted personalized features; The personalized features occupy a preset weight proportion in subsequent scheduling planning and can be dynamically adjusted according to the actual operation feedback of the doctor.

6. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 1, characterized in that: The doctor desktop operation trajectory prediction processing includes the following steps: Multi-modal information analysis: The multi-modal large model receives screen interface images, real-time doctor operations, and medical context feature vectors, and analyzes the output current operation intention; Single-step action prediction: Combine the current operation intention and medical context to output the next atomic operation and confidence; Continuous operation trajectory planning: Based on the current operation intention and the predicted single-step action, the future multi-step operation trajectory is optimized and planned in the rolling time domain, and medical rules are integrated into the planning process for verification.

7. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 6, characterized in that: The rolling time domain optimization planning specifically includes: Taking the current time as the starting point, predicting the future k-step trajectory, and re-optimizing the subsequent trajectory after each step is executed; In the optimization process, the dynamic programming algorithm is used to minimize the trajectory deviation, and the cost function based on state and operation action is calculated; Medical rule verification is realized through a rule engine, which automatically corrects when the trajectory lacks key steps.

8. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 1, characterized in that: The diagnosis and treatment task planning and scheduling processing includes the following steps: Task extraction and modeling: Extract the tasks to be processed from the doctor's desktop window and to-do reminders, and label the task attribute information, including urgency, complexity, and associated patient type; Priority sorting: Based on the task attribute information, the priority score of each task is calculated using a multi-factor weighted scoring method; Time planning and scheduling: According to the task priority score, and combining the task time consumption of operation trajectory prediction and the doctor's work gap, a visual scheduling table that supports real-time adjustment is generated.

9. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 8, characterized in that: The multi-factor weighted scoring method for calculating the priority score of each task includes the following specific steps: Scoring factor definition and weight configuration: Establish a scoring system composed of three core factors: task urgency, task complexity, and doctor operation habit preference, and configure an initial weight for each factor that can be dynamically adjusted; Urgency score mapping: Based on the pre-defined medical rule library, the urgency level in the task attribute is mapped to a standardized urgency score value, where emergency tasks automatically get the highest level score; Complexity score calculation: Analyze the clinical data features related to the task, and calculate the complexity score based on the number of diagnosis and treatment operation steps, the number of involved complications, and the required medical resource level; Habit preference score extraction: Quantitatively extract the operation habit preference score from the doctor's personalized context features, including weighted statistics of historical completion efficiency and dependence on common templates for specific task types; Dynamic weighted fusion: Dynamically adjust the weight proportion of the three types of factors according to the real-time diagnosis and treatment scene demand, and use linear weighted summation algorithm to generate the final task priority comprehensive score, the calculation formula is: ; wherein is a comprehensive priority score, is an urgency weight, is an urgency score, is a complexity weight, is a complexity score, is a habit weight, is a habit score; wherein the weight factors satisfy a normalization constraint ; Score result verification and correction: compare the calculated comprehensive score with the historical priority threshold, and trigger the rule engine for automatic correction when a score logic conflict is detected.

10. The multi-modal large model based desktop operation trajectory prediction and scheduling planning method according to claim 1, characterized in that: The dynamic optimization and feedback closed loop processing includes the following steps: Real-time feedback adjustment: based on the execution effect of the task planning scheduling data, the doctor corrects the predicted trajectory and scheduling result through voice instructions and interface buttons, and the corrected data is fed back to the multi-modal large model for incremental training to generate updated model parameters and feedback logs; Periodic optimization: based on the updated model parameters and feedback logs, the prediction accuracy and scheduling adaptation rate are extracted from the multi-modal large model every week, and the gradient descent algorithm is used to optimize the model weight to minimize the loss function, completing the periodic adjustment of the model parameters.

Images