Interaction control method based on large model driving

Abstract

The invention relates to the technical field of computers, and discloses an interaction control method based on large model driving, which comprises the following steps: acquiring task data issued by external equipment connected with a target computer, generating a corresponding initial operation instruction for controlling a keyboard and a mouse connected with the target computer, and controlling the target computer to execute a corresponding initial operation; after the preliminary operation is executed, a display signal on a target computer screen is obtained, key information is extracted from the display signal, the key information is input into the target large model, and a re-operation instruction is generated; and controlling the target computer to execute corresponding re-operation, after the re-operation is executed, judging whether the tasks of the task data are completely executed or not based on the re-display signal, if not, continuing real-time analysis, and controlling the target computer to execute corresponding operation based on the operation instruction corresponding to the keyboard and the mouse until all the tasks of the task data are completely executed. According to the invention, the operation efficiency and flexibility of man-machine interaction can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of computers, and in particular to an interactive control method based on large model driving. Background Art

[0002] With the rapid development of computer technology, human-computer interaction is also evolving. Traditional interaction methods mainly rely on keyboards and mice. Although this method is simple and intuitive, it is inefficient in some scenarios, especially in tasks that require long-term repetitive operations. In recent years, breakthroughs in computer vision and deep learning technology have enabled machines to understand and process complex visual information more accurately, making it possible to achieve higher levels of automation.

[0003] Currently, there are some automation tools on the market, such as macro recording software and robotic process automation (RPA) tools, but they usually require software to be installed on the target computer and have relatively limited functions, making it difficult to cope with complex and changing task requirements.

[0004] From the above, we can see that how to improve the operational efficiency and flexibility of human-computer interaction still needs to be solved. Summary of the invention

[0005] In order to improve the operational efficiency and flexibility of human-computer interaction, the present application provides an interactive control method based on large model driving.

[0006] In the first aspect, the present application provides an interactive control method based on large model driving, which adopts the following technical solutions:

[0007] An interactive control method based on large model driving, comprising: determining a corresponding target computer, obtaining task data issued by an external device connected to the target computer, generating preliminary operation instructions corresponding to a keyboard and mouse connected to the target computer based on the task data, and sending the preliminary operation instructions corresponding to the keyboard and mouse to the target computer, controlling the target computer to perform a corresponding preliminary operation based on the preliminary operation instructions corresponding to the keyboard and mouse; after performing the preliminary operation, obtaining a display signal on the screen of the target computer, calling a pre-set image processing unit, controlling the image processing unit to perform real-time analysis on the display signal, and extracting corresponding key information, calling a large image in an external device Model set, determine the corresponding target large model in the large model set based on the key information, input the key information into the target large model, generate corresponding re-operation instructions for keyboard and mouse; control the target computer to perform corresponding re-operation based on the corresponding re-operation instructions for keyboard and mouse, after performing the re-operation, obtain a re-display signal on the screen of the target computer, and judge whether the tasks corresponding to the task data have been fully executed based on the re-display signal, if not, call the image processing unit to continue real-time analysis of the re-display signal, and control the target computer to perform corresponding operations based on the corresponding operation instructions for keyboard and mouse until all tasks corresponding to the task data are completed.

[0008] By adopting the above technical solution, the target computer is determined and task data is obtained, a preliminary operation instruction is generated and a preliminary operation is performed, and then the screen display signal is analyzed in real time by the image processing unit, key information is extracted, and the large model is called to generate a re-operation instruction to control the target computer to perform the re-operation; during the whole process, the screen display signal is continuously monitored, and the operation instructions are dynamically generated and executed until all tasks corresponding to the task data are completed. Through intelligent analysis and dynamic adjustment, each step of the operation is ensured to be accurately executed, which improves the operational efficiency and flexibility of human-computer interaction, reduces the need for manual intervention, and enables users to complete tasks more efficiently and accurately.

[0009] Optionally, in the process of the target computer performing corresponding operations based on the corresponding operation instructions of the keyboard and mouse, including the initial operation and the re-operation, the method also includes: obtaining the corresponding operating status and operation results of the target computer based on the target computer executing the operation; inputting the operating status and the operation results into a pre-set abnormality detection model for identification, and obtaining the corresponding detection results; if there is an abnormality in the detection result, automatically pausing the current automation process, determining the corresponding abnormality severity level, and generating a corresponding abnormality record, wherein the abnormality record includes the time, type, cause and related operation log of the abnormality.

[0010] By adopting the above technical solution, through anomaly detection, after determining the severity level of the anomaly, an anomaly record containing the time, type, cause and related operation log of the anomaly can be generated; abnormal situations can be discovered and handled in a timely manner, ensuring the stability and reliability of the system, preventing data corruption or system crashes caused by erroneous operations, thereby improving the overall security and operational efficiency of human-computer interaction.

[0011] Optionally, the method also includes: regularly obtaining historical operation records of users, and collecting corresponding feedback data from users, wherein the historical operation records include operation steps, operation time, and operation results, and the feedback data include user satisfaction evaluations on the operation results, suggested areas for improvement, and problems encountered; fusing the historical operation records and user feedback data to form a corresponding comprehensive data set, analyzing the comprehensive data set to identify operation recognition results in user operations, wherein the operation recognition results include common patterns, error points, and optimization points; generating a parameter adjustment strategy corresponding to the target large model based on the operation recognition results, and the parameter adjustment strategy includes adjusting the model's weights and biases, modifying the model's hyperparameters, and increasing or decreasing the number of model layers.

[0012] By adopting the above technical solution, by regularly obtaining the user's historical operation records and feedback data, forming a comprehensive data set and analyzing it, identifying common patterns, error points and optimization points in user operations, and then generating a parameter adjustment strategy for the target large model, the above steps can continuously optimize the performance of the large model, improve the accuracy and efficiency of operations, and ensure that the system better adapts to the user's operating habits and needs, thereby improving the user experience and the overall performance of the system.

[0013] Optionally, in the process of controlling the control image processing unit to perform real-time analysis on the display signal, the method also includes: identifying the type of task currently being performed by the user based on the real-time analysis operation, and generating a corresponding current task description based on the task type; retrieving the user's historical operation records, matching the current task description with the tasks in the historical operation records, and selecting the best operation guide based on the matching results; generating a corresponding recommended operation path based on the best operation guide, and displaying the recommended operation path to the user through a user interface, wherein the recommended operation path includes specific steps, operation sequence, and precautions.

[0014] By adopting the above technical solution, the user's current operation is analyzed in real time, the type of task being performed is identified, and the current task description is generated. The system matches the current task with the user's historical operation records, selects the best operation guide, and generates a recommended operation path, which is displayed to the user through the user interface. This mechanism can provide personalized operation suggestions, help users complete tasks more efficiently and accurately, and improve user experience and operation efficiency.

[0015] Optionally, in the process of controlling the control image processing unit to perform real-time analysis on the display signal, the method also includes: identifying the user's current context information based on the real-time analysis operation, and generating a corresponding current context description; retrieving the user's historical operation records, matching the current context description with the context in the historical operation records, and predicting the user's corresponding next operation intention based on the matching result; generating a corresponding next operation instruction based on the next operation intention, sending the next operation instruction to the user, and providing an option for the user to choose whether to accept. If the user confirms to accept the predicted next operation instruction, the cached next operation instruction is sent to the target computer for operation.

[0016] By adopting the above technical solution, the current context information of the user is analyzed in real time, the current context description is generated, and it is matched with the user's historical operation records to predict the user's next operation intention. The system generates the corresponding next operation instruction based on the predicted operation intention, and provides options for the user to choose whether to accept. If the user confirms to accept, the system sends the cached next operation instruction to the target computer for operation. This mechanism can prepare and execute possible user operations in advance, improve the efficiency and fluency of operations, and reduce the user's waiting time.

[0017] Optionally, after the operation based on real-time analysis identifies the type of task currently being performed by the user, the method further includes: calling predefined priority evaluation rules, and assigning task priorities to corresponding tasks based on the task type and the priority evaluation rules; monitoring the resource usage of the target computer in real time, and evaluating the resource availability of the current system, wherein the resource usage includes CPU usage, memory usage, and disk read and write speeds; and adjusting the execution order of all tasks to be executed based on the task priority and the resource availability.

[0018] By adopting the above technical solution, the resource usage of the target computer is monitored in real time by identifying the type of task currently being executed by the user and assigning task priority, and the resource availability of the system is evaluated. Based on task priority and resource availability, the execution order of all pending tasks is dynamically adjusted. This mechanism can ensure that high-priority tasks are executed first, reasonably allocate system resources, and improve the overall operating efficiency and stability of the system.

[0019] In a second aspect, the present application provides an interactive control device based on large model drive, which adopts the following technical solution:

[0020] An interactive control device based on large model driving, comprising:

[0021] A preliminary operation instruction generating module determines a corresponding target computer, obtains task data issued by an external device connected to the target computer, generates preliminary operation instructions corresponding to a keyboard and mouse connected to the target computer based on the task data, and sends the preliminary operation instructions corresponding to the keyboard and mouse to the target computer, controlling the target computer to perform a corresponding preliminary operation based on the preliminary operation instructions corresponding to the keyboard and mouse;

[0022] A re-operation instruction generating module, after performing the preliminary operation, obtains the display signal on the target computer screen, calls a preset image processing unit, controls the image processing unit to perform real-time analysis on the display signal, and extracts corresponding key information, calls a large model set in an external device, determines a corresponding target large model in the large model set based on the key information, inputs the key information into the target large model, and generates a corresponding re-operation instruction for the keyboard and mouse;

[0023] The task judgment module controls the target computer to perform a corresponding re-operation based on the corresponding re-operation instructions of the keyboard and mouse. After performing the re-operation, a re-display signal on the screen of the target computer is obtained, and the re-display signal is used to judge whether all tasks corresponding to the task data have been completed. If not, the image processing unit is called to continue to perform real-time analysis on the re-display signal, and the target computer is controlled to perform a corresponding operation based on the corresponding operation instructions of the keyboard and mouse until all tasks corresponding to the task data are completed.

[0024] In a third aspect, the present application provides an interactive control method based on large model drive, which adopts the following technical solutions:

[0025] An interactive control method based on large model driving comprises a processor, wherein the program of any one of the interactive control methods based on large model driving described above is run in the processor.

[0026] In a fourth aspect, the present application provides a storage medium, which adopts the following technical solution:

[0027] A storage medium stores a program of any one of the above-mentioned interactive control methods based on large model drive.

[0028] In summary, the present application includes at least one of the following beneficial technical effects:

[0029] By continuously monitoring the screen display signals, dynamically generating and executing operating instructions until all tasks corresponding to the task data are completed, and through intelligent analysis and dynamic adjustment, ensuring that each step of the operation can be executed accurately, the operational efficiency and flexibility of human-computer interaction are improved, the need for manual intervention is reduced, and users can complete tasks more efficiently and accurately.

[0030] Through the anomaly detection mechanism, abnormal situations can be discovered and handled in a timely manner, and detailed anomaly records can be generated to prevent data corruption or system crashes caused by erroneous operations, thereby improving the overall security and operational efficiency of human-computer interaction. By regularly obtaining users' historical operation records and feedback data, a comprehensive data set is formed and analyzed to identify common patterns, error points and optimization points in user operations, continuously optimize the performance of large models, improve the accuracy and efficiency of operations, and ensure that the system better adapts to users' operating habits and needs.

[0031] By analyzing the user's current operations in real time, identifying the type of task being performed, and generating recommended operation paths, it helps users complete tasks more efficiently and accurately. By analyzing the user's current context information in real time, it predicts the user's next operation intention, generates corresponding operation instructions, and provides options for users to choose whether to accept, thereby improving the efficiency and smoothness of operations and reducing user waiting time. It can also identify the type of task the user is currently performing and assign task priorities, monitor the resource usage of the target computer in real time, dynamically adjust the execution order of all tasks to be executed, ensure that high-priority tasks are executed first, reasonably allocate system resources, and improve the overall operation efficiency and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The present invention is a flowchart of an interactive control method based on large model driving according to an exemplary embodiment.

[0033] Figure 2 It is a structural block diagram of an interactive control method and device based on large model driving according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings.

[0035] In the description of this specification, the description with reference to the terms "certain embodiments", "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0036] The present application embodiment discloses an interactive control method based on large model driving, referring to Figure 1 ,include:

[0037] S100, determine the corresponding target computer, obtain the task data issued by the external device connected to the target computer, generate the corresponding preliminary operation instructions for controlling the keyboard and mouse connected to the target computer based on the task data, and send the corresponding preliminary operation instructions for the keyboard and mouse to the target computer, and control the target computer to perform the corresponding preliminary operation based on the corresponding preliminary operation instructions for the keyboard and mouse.

[0038] Among them, first of all, it is necessary to clearly identify the target computer that needs to be automated controlled. This can be a specific desktop computer, laptop computer or other computing device with an external display interface. In addition to the USB interface, the target computer also has wireless keyboard and mouse interfaces such as Bluetooth and Star Flash, as well as wired keyboard and mouse interfaces; through external devices (such as HDMI capture card, VGA capture card, USB camera, etc.) connected to the display output interface (such as HDMI, VGA, DisplayPort, etc.) of the target computer. It should be pointed out that these external devices are used to collect display signals on the target computer screen, and corresponding tasks can be issued through external devices.

[0039] It should be pointed out here that task data can come from external devices connected to the target computer, such as mobile devices, servers, user interfaces, etc. These devices can send task data to the control system through a network or physical interface, and the task data includes detailed information about the task, such as task type, operation steps, target object, expected results, etc. The control system receives and parses the task data sent by external devices to ensure the integrity and accuracy of the data.

[0040] Thirdly, according to the received task data, the specific requirements and operation steps of the task are parsed, and then based on the parsed task requirements, the corresponding preliminary operation instructions for the keyboard and mouse to control the target computer are generated. These preliminary operation instructions may include keyboard input, mouse clicks, dragging and other operations; in addition, it is necessary to ensure that the generated preliminary operation instructions meet the input format requirements of the target computer and can be correctly recognized and executed by the target computer.

[0041] It should be pointed out here that the external device in the embodiment of the present application has a high-computing power TPU neural network hardware inference unit, which can be used for the calculation of image inference algorithms and large model algorithms. The high-computing power TPU neural network hardware inference unit is a high-performance hardware accelerator designed specifically for accelerating machine learning tasks. It has extremely high computing power and can quickly process large-scale matrix operations, and is particularly suitable for reasoning tasks of deep learning models. TPU provides low latency and high throughput, suitable for real-time application scenarios, and has a high energy efficiency ratio, and can provide excellent computing performance at low power consumption. In the embodiment of the present application, TPU is used to analyze the display signal on the target computer screen in real time, extract key information, and generate and output corresponding keyboard and mouse commands, operate computer software, and automatically, accurately and efficiently complete a series of work tasks.

[0042] In addition, this external device is equipped with a MIC, a speaker, and a camera, and can interact directly with the operator through voice and image recognition. The operator can issue commands through voice, intervene in the intermediate processing, and confirm the processing results. The camera can also collect images of the operator's eye movements, expressions, gestures, behaviors, etc. in real time, and recognize the operator's behavior and instructions through the TPU algorithm model, and work according to the instructions. These functions enable the system to achieve more natural and efficient user interaction and improve the accuracy and flexibility of task execution.

[0043] Specific preliminary operation instructions are generated based on the task data to ensure that the system can perform preliminary operations, and the generated preliminary operation instructions accurately reflect the requirements of the task, thereby improving the accuracy of the operation.

[0044] Finally, the generated preliminary operation instructions are sent to the target computer through an external device (such as a USB adapter, wireless transmitter, etc.), and after receiving the preliminary operation instructions, the target computer performs corresponding operations, which may include opening applications, entering text, clicking buttons, etc., and the system monitors the execution status of the target computer in real time to ensure that the preliminary operation is performed as expected.

[0045] That is to say, the control system sends the preliminary keyboard and mouse operation instructions to the target computer through external devices (such as USB adapters, wireless transmitters, etc.). These instructions simulate real keyboard and mouse operations, and the target computer receives and executes these instructions; the target computer performs corresponding operations based on the received operation instructions, thereby realizing automatic control. For example, if the operation instruction is to click a button, the target computer will simulate the user's action of clicking the button.

[0046] By sending preliminary operation instructions, automated control is achieved, reducing the need for human intervention, while ensuring that the preliminary operations are performed as expected, providing a basis for subsequent further operations.

[0047] S110, after performing preliminary operations, obtain the display signal on the target computer screen, call the pre-set image processing unit, control the image processing unit to perform real-time analysis on the display signal, and extract the corresponding key information, call the large model set in the external device, determine the corresponding target large model in the large model set based on the key information, input the key information into the target large model, and generate the corresponding re-operation instructions for the keyboard and mouse.

[0048] Among them, after performing preliminary operations, the display signal on the target computer screen is collected in real time through an external device connected to the target computer (such as an HDMI capture card, USB camera, etc.), and the collected display signal is transmitted to the control system to ensure the real-time and integrity of the data. The collected display signal can be a video stream or an image frame, depending on the type and configuration of the acquisition device.

[0049] Then, the control system calls a pre-set image processing unit, which usually includes image processing hardware (such as image TPU) and corresponding software algorithms, and is mainly used to process and analyze the collected display signals; and the image processing unit performs real-time analysis on the collected display signals, which includes but is not limited to image segmentation, feature extraction, object recognition and other operations, and the purpose of the analysis is to extract key information from the display signals, which will be used for subsequent automated operations.

[0050] Finally, key information is extracted from the analysis results. The key information can be specific elements on the screen (such as buttons, icons, text, etc.) or higher-level information (such as task status, location of the operation object, etc.).

[0051] Based on the above process, through efficient image processing technology, the key elements on the screen can be accurately identified and the key information on the screen can be quickly extracted to ensure that the system can respond to the user's operational needs in a timely manner.

[0052] Then, the control system calls up a collection of pre-trained large models. It should be noted that these large models can be locally deployed or remote large models connected via a network. The large model collection contains multiple models for different tasks and scenarios. Each model is specially trained to process specific types of inputs and generate corresponding outputs.

[0053] Then, based on the extracted key information, the control system determines the most suitable model in the large model set as the target large model. This can be achieved by matching the characteristics of the key information with the input requirements of the large model. For example, if the key information is an interface element of a specific application, the control system will select a large model trained specifically for that application.

[0054] By selecting the appropriate big model, the system can flexibly respond to different tasks and scenarios and provide personalized operation suggestions, which can ensure that the selected big model can accurately process the current key information and improve the reliability of generating operation instructions.

[0055] Finally, the extracted key information is input into the target large model. The key information is used as the input of the model to generate corresponding re-operation instructions, which may include but are not limited to operations such as clicking, dragging, and inputting text.

[0056] By utilizing pre-trained large models and selecting appropriate large models, the system can flexibly respond to different tasks and scenarios, and generate precise operation instructions based on the extracted key information, which can improve the accuracy and reliability of operations.

[0057] S120, controlling the target computer to perform a corresponding re-operation based on the corresponding re-operation instructions of the keyboard and mouse, and after performing the re-operation, obtaining a re-display signal on the target computer screen, and judging whether all tasks corresponding to the task data have been completed based on the re-display signal, if not, calling the image processing unit to continue real-time analysis of the re-display signal, and controlling the target computer to perform a corresponding operation based on the corresponding operation instructions of the keyboard and mouse until all tasks corresponding to the task data have been completed.

[0058] The generated keyboard and mouse re-operation instructions are sent to the target computer through an external device (such as a USB adapter, wireless transmitter, etc.), and the target computer performs the corresponding operations after receiving the re-operation instructions. These operations may include clicking a button, dragging a window, inputting text, etc., and the system monitors the execution status of the target computer in real time to ensure that the re-operation is performed as expected.

[0059] After the re-operation is performed, the re-display signal on the target computer screen is collected in real time through an external device connected to the target computer (such as an HDMI capture card, USB camera, etc.), and the collected re-display signal is transmitted to the control system to ensure the real-time and integrity of the data.

[0060] At this time, it is necessary to call the image processing unit to perform real-time analysis on the re-display signal to extract the corresponding key information. It is also necessary to judge whether the tasks corresponding to the task data have been fully executed based on the extracted key information. The judgment basis may include task completion signs (such as specific interface elements, prompt information, etc.). If the task has been completed, the system records the task completion status; if the task has not been completed, proceed to the next step. By analyzing the re-display signal in real time, it is possible to accurately judge whether the task has been completed, ensuring that the system can adjust the operation strategy in a timely manner.

[0061] If the task is not completed, the image processing unit is called to continue real-time analysis of the display signal again, extract new key information, and then based on the new key information, the large model is called to generate new keyboard and mouse operation instructions, and the generated new operation instructions are sent to the target computer through an external device, and the target computer executes the new operation instructions. The system continues to monitor the display signal of the target computer and repeats the above steps until all tasks corresponding to the task data are completed.

[0062] Based on the above technical solution, the following corresponding effects can be achieved:

[0063] Automated control: By generating and sending operation instructions, automated control is achieved, reducing the need for human intervention; automated control improves the efficiency of task execution and ensures that each operation is carried out as expected.

[0064] Real-time monitoring: collect the display signal of the target computer in real time through external devices to ensure the real-time and integrity of the data; call the image processing unit to perform real-time analysis on the display signal, extract key information, and ensure that the system can respond to the user's operation needs in a timely manner.

[0065] Accurate identification: Use advanced image processing technology to accurately identify key elements on the screen and improve the accuracy and reliability of operations; based on the extracted key information, select the most appropriate large model to ensure that the generated operation instructions are accurate and reliable.

[0066] Dynamic adjustment: Based on the results of real-time analysis, new operation instructions are dynamically generated to ensure that each step of the operation can be accurately executed; by cyclically judging the completion status of tasks, it is ensured that all tasks corresponding to the task data are completed, thereby improving the completion rate and accuracy of tasks.

[0067] Task completion: Through real-time analysis of display signals, accurately determine whether the task is completed, ensuring that the system can adjust the operation strategy in time; record the task completion status to provide a basis for subsequent task management and reporting.

[0068] In addition, the above technical solution actually includes three parts: the external device, the target computer is the host, and the target computer screen. Similarly, if the target computer is an integrated body and screen, such as a laptop computer, the above technical solution is also applicable.

[0069] In the solution of the embodiment of the present application, in the process of the target computer performing a corresponding operation based on the corresponding operation instruction of the keyboard and mouse, wherein the execution of the corresponding operation includes the first execution of the operation and the re-execution of the operation, the method further includes:

[0070] S1201, obtaining the operating status and operation result corresponding to the target computer based on the target computer executing the operation.

[0071] Among them, during the process of the target computer executing the operation instruction, the system monitors the operating status and operation results of the target computer in real time, and the monitoring content includes but is not limited to the changes in the screen display content, system logs, the execution results of the operation instructions, etc. At the same time, the system uses the image processing unit and the large model to analyze the screen display content in real time, identify the interface changes after the operation, and record the system logs and the execution results of the operation instructions.

[0072] S1202, input the running status and operation results into a preset anomaly detection model for identification, and obtain corresponding detection results.

[0073] Among them, the obtained running status and operation results are organized into structured data and prepared to be input into the anomaly detection model. The system calls the pre-set anomaly detection model. The anomaly detection model can be a machine learning-based model, such as decision trees, random forests, neural networks, etc., or a rule-based model.

[0074] The anomaly detection model analyzes the input operation status and operation results to identify whether there are any abnormal situations, where the abnormal situations may include but are not limited to the operation instructions not being executed as expected, the screen display content not being consistent with expectations, the system response timeout or no response, the user manually intervenes or cancels the operation, etc. The anomaly detection model outputs the detection results, indicating whether there is an abnormality and its type.

[0075] S1203: If there is an abnormality in the detection result, the current automated process is automatically suspended, the corresponding abnormality severity level is determined, and a corresponding abnormality record is generated.

[0076] Among them, the system determines whether there is an abnormal situation based on the output results of the anomaly detection model. If there is an abnormality, the system automatically suspends the current automation process to prevent further erroneous operations or data damage. And according to the type and scope of the abnormality, the severity level of the abnormality is determined, such as minor abnormality, medium abnormality and severe abnormality. At the same time, the system generates detailed abnormality records, which include the time, type, cause and related operation logs of the abnormality. The abnormality records can be stored in the log file for subsequent analysis and troubleshooting.

[0077] Based on executing the above steps S1201 to S1203, first of all, it can be ensured that the system can comprehensively monitor the running status and operation results of the target computer and discover potential problems in time; and when an abnormality is detected, the current automation process is automatically suspended to prevent data damage or system crash caused by erroneous operation; and different processing measures can be taken according to the severity level of the abnormality to ensure the stability and reliability of the system.

[0078] In the solution of the embodiment of the present application, the method further includes:

[0079] S121, regularly obtain the user's historical operation records and collect the user's corresponding feedback data.

[0080] The system collects historical operation records of users on a regular basis (such as daily or weekly). Historical operation records include detailed information such as operation steps, operation time, and operation results, which can be stored in log files or databases. At the same time, the system provides multiple user feedback channels, such as feedback buttons in the user interface, emails, online forms, etc., to facilitate users to submit feedback information such as satisfaction evaluation of operation results, suggestions for improvement, and problems encountered.

[0081] S122, integrating the historical operation records and the user feedback data to form a corresponding comprehensive data set, analyzing the comprehensive data set, and identifying the operation recognition results in the user operation.

[0082] Among them, historical operation records and user feedback data are integrated to form a comprehensive data set. The comprehensive data set includes multi-dimensional information such as operation steps, operation time, operation results, user satisfaction evaluation, suggested areas for improvement, and problems encountered. Here, the comprehensive data set needs to be cleaned to remove invalid or erroneous data to ensure the quality and accuracy of the data.

[0083] Use data analysis tools and algorithms to analyze comprehensive data sets. The purpose of the analysis is to identify common patterns, error points, and optimization points in user operations. Common pattern recognition: Identify common behavioral patterns of users during operation, such as commonly used shortcut key combinations, frequently accessed interfaces, etc., which helps to optimize the design and operation process of the system; Error point recognition: Identify common error points in the user's operation process, such as misoperation, operation failure, etc., which helps to fix problems in a timely manner and improve the stability of the system; Optimization point recognition: Identify operation points that can be optimized, such as simplifying operation steps, improving operation efficiency, etc., which helps to continuously improve the performance of the system and user experience.

[0084] S123, generating a parameter adjustment strategy corresponding to the target large model based on the operation recognition result.

[0085] Among them, the parameter adjustment strategy includes adjusting the weight and bias of the model, modifying the hyperparameters of the model, and increasing or decreasing the number of layers of the model. According to the operation recognition results, the parameter adjustment strategy of the target large model is generated. The parameter adjustment strategy may include but is not limited to adjusting the weight and bias of the model, modifying the hyperparameters of the model, increasing or decreasing the number of layers of the model, etc.

[0086] Specifically, adjust the model weights and biases: adjust the model weights and biases according to the identified common patterns and optimization points to better fit the user operation patterns; modify the model hyperparameters: modify the model hyperparameters, such as learning rate, batch size, etc., according to the identified error points and optimization points to improve the training effect of the model; increase or decrease the number of model layers: increase or decrease the number of model layers according to the identified operation complexity to adapt to tasks of different complexities.

[0087] Based on executing the above steps S121 to S123, by continuously optimizing the parameters of the large model, the accuracy and efficiency of the operation are improved, ensuring that the system can better adapt to the user's operating habits and needs, thereby improving the user experience and the overall performance of the system.

[0088] In addition, in the process of controlling the image processing unit to perform real-time analysis on the display signal, the method further includes:

[0089] S124, identifying the type of task currently being performed by the user based on the real-time analysis operation, and generating a corresponding current task description based on the task type.

[0090] The system uses an image processing unit to analyze the display signal on the target computer screen in real time, extract the currently displayed content and the user's ongoing operations, and identify the type of task the user is currently performing based on the extracted content and operations. The task type can be file processing, data entry, graphic design, programming development, etc. Then, based on the identified task type, a detailed description of the current task is generated, including the task's objectives, key operation steps, and operation objects.

[0091] S125, retrieve the user's historical operation records, match the current task description with the tasks in the historical operation records, and select the best operation guide according to the matching result.

[0092] The system retrieves the user's historical operation records and matches the current task description with the tasks in the historical operation records. The matching basis may include task type, operation steps, operation objects, etc. According to the matching results, the optimal operation guide in the historical operation records is selected. The optimal operation guide can be the operation steps and methods that the user has successfully completed similar tasks in the past. By matching and selecting the best operation guide, it ensures that the recommended operation path is optimal and reduces the user's trial and error costs.

[0093] S126: Generate a corresponding recommended operation path based on the optimal operation guide, and display the recommended operation path to the user through a user interface.

[0094] Among them, according to the selected best operation guide, a recommended operation path is generated, and the recommended operation path includes specific steps, operation sequence, precautions, etc.; the recommended operation path is displayed to the user through a user interface, and the user interface can be a graphical interface, text prompts, voice prompts, etc., to ensure that the user can clearly see the recommended operation path. The system can also provide a user confirmation function, and the user can view the recommended operation path and choose whether to accept it. If not, the user can choose to operate manually or provide new feedback.

[0095] Based on executing the above steps S124 to S126, a personalized recommended operation path is generated through real-time analysis and historical data matching to help users complete tasks more efficiently and accurately, thereby improving user experience and operational efficiency; a user confirmation function is also provided to enhance user participation and control, thereby improving user satisfaction.

[0096] In addition, in the process of controlling the image processing unit to perform real-time analysis on the display signal, in order to further help the user complete the task more efficiently and accurately, the method also includes:

[0097] S127, identifying the user's current context information based on the real-time analysis operation, and generating a corresponding current context description.

[0098] Among them, similarly, the system first analyzes the display signal on the target computer screen in real time through the image processing unit, extracts the currently displayed content and the user's ongoing operations, and then identifies the user's current context information based on the extracted content and operations. It should be pointed out here that the context information may include the status of the current application, the specific operations performed by the user, key elements on the screen, etc., and based on the identified context information, generates a detailed description of the current context, including the status of the current task, the location of the operation object, the user's current operation stage, etc.

[0099] S128, retrieve the user's historical operation records, match the current context description with the context in the historical operation records, and predict the user's corresponding next operation intention based on the matching result.

[0100] Among them, the system retrieves the user's historical operation records, which include detailed information such as operation steps, operation time, operation results, context information, etc., and then matches the current context description with the context in the historical operation records. The matching basis may include context information, operation steps, operation objects, etc.

[0101] Based on the matching results, the system can predict the user's next operation intention. The basis for the prediction can be the user behavior patterns and common operation sequences under similar contexts in historical operation records. Through matching and prediction, the user's next operation intention can be identified in advance.

[0102] S129, generating a corresponding next operation instruction based on the next operation intention, sending the next operation instruction to the user, and providing an option for the user to choose whether to accept. If the user confirms to accept the predicted next operation instruction, the cached next operation instruction is sent to the target computer for operation.

[0103] Among them, the corresponding next operation instructions are generated according to the predicted next operation intention. These instructions may include keyboard and mouse operations, such as clicking, dragging, inputting text, etc. Displaying the operation instructions through the user interface or voice prompts can ensure that the user can clearly understand and confirm the recommended operation steps.

[0104] At the same time, the system provides a user confirmation function. In order to enhance the user's sense of participation and control, if the user confirms and accepts, the system will automatically execute the operation instructions, reducing the user's waiting time and operation burden, and improving the smoothness and efficiency of the operation.

[0105] Through real-time analysis and historical data matching, the user's next operation intention is predicted and corresponding operation instructions are generated to help users complete tasks more efficiently and accurately, improving user experience and operational efficiency.

[0106] Finally, after identifying the type of task currently being performed by the user based on the real-time analysis operation, the method further includes:

[0107] S1241, retrieve a predefined priority evaluation rule, and assign a task priority to the corresponding task based on the task type and the priority evaluation rule.

[0108] The system retrieves the predefined priority evaluation rules, which can be multi-dimensional evaluation criteria based on task type, task importance, task urgency, etc. The system has identified the type of task the user is currently performing through real-time analysis, and can then assign a priority to the current task based on the task type and predefined priority evaluation rules. The priority can be a numerical value that indicates the urgency and importance of the task. According to the task type and evaluation rules, each task is assigned an appropriate priority to ensure that high-priority tasks are given priority.

[0109] S1242, monitor the resource usage of the target computer in real time and evaluate the resource availability of the current system.

[0110] Among them, the system can monitor the resource usage of the target computer in real time, including key indicators such as CPU usage, memory usage, disk read and write speed, and evaluate the resource availability of the current system based on the monitored resource usage. It should be pointed out that the resource availability evaluation can include but is not limited to the following: CPU usage: evaluate the CPU load to determine whether there are sufficient computing resources; memory usage: evaluate memory usage to determine whether there is sufficient memory space; disk read and write speed: evaluate the disk read and write speed to determine whether there is sufficient I / O bandwidth. By evaluating resource availability, ensure that the system can reasonably allocate resources and avoid resource overload.

[0111] S1243, adjusting the execution order of all tasks to be executed based on task priorities and resource availability.

[0112] Among them, the system maintains a task queue, sorts all tasks to be executed by priority, with high-priority tasks in front and low-priority tasks in the back; then dynamically adjusts the execution order of tasks based on task priority and resource availability. If current resources are tight, the system can give priority to high-priority tasks and delay the execution of low-priority tasks.

[0113] Dynamically allocate resources based on the resource requirements of the task and the current resource availability. For example, for compute-intensive tasks, the system can allocate more CPU resources; for I / O-intensive tasks, the system can allocate more disk bandwidth.

[0114] By dynamically adjusting the execution order of tasks and resource allocation, we can ensure efficient use of resources, avoid resource waste and bottleneck problems, ensure that high-priority tasks are executed first, and improve the processing speed and efficiency of key tasks; thus, we can reasonably allocate resources, ensure the stability and reliability of the system, and prevent system crashes or performance degradation caused by resource overload.

[0115] Finally, for the actual application examples of the solutions in the embodiments of this application:

[0116] Assume that a user needs to fill in personal information in an online form, including name, address, phone number, etc.; usually, the user needs to manually click on each field in the form and enter the corresponding information. Now, the external device of the solution in the embodiment of the present application is used to automate this process through image reasoning algorithm and large model algorithm.

[0117] Determine the target computer and task data: the user issues task data through an external device (such as a mobile device, server, user interface, etc.), and the task data includes the URL of the form, the information that needs to be filled in, etc.; the system determines the target computer and connects to the display output interface of the target computer through an external device (such as an HDMI capture card, USB camera, etc.); the system parses the task data and generates preliminary keyboard and mouse operation instructions for controlling the target computer, such as opening a browser and navigating to a specified form page; then the preliminary operation instructions are sent to the target computer through an external device (such as a USB adapter, wireless transmitter, etc.), and the target computer performs the corresponding operation to open the form page.

[0118] After the form page is opened, the external device collects the display signal on the target computer screen in real time through the camera; at this time, the image processing unit analyzes the display signal in real time and extracts key information, such as the position and label of each field in the form. The system calls the pre-trained large model set, determines the most suitable target large model based on the extracted key information, and then inputs the key information into the target large model to generate the corresponding re-operation instructions of the keyboard and mouse, such as clicking the form field, entering text, etc.; and the re-operation instructions are sent to the target computer through the external device, and the target computer performs the corresponding operation and automatically fills in the form fields.

[0119] The system continues to collect the display signal on the target computer screen in real time, calls the image processing unit to perform real-time analysis on the display signal, extracts new key information, and determines whether the task is completed based on the extracted key information, such as checking whether the form submission button is clicked. If the task is not completed, the system continues to generate new operation instructions and execute them until the task is completed.

[0120] Compare the above process with manual operation in various aspects:

[0121] 1. Click on the form fields: Whether it is manual operation or automated operation, you need to click on each field in the form; automated operation simulates the user's click action through the generated keyboard and mouse commands to ensure that the click position and timing are consistent with manual operation.

[0122] 2. Input text: During manual operation, users will enter text in the form field; automated operation simulates the process of users entering text through generated keyboard and mouse commands to ensure that the input content and order are consistent with manual operation.

[0123] 3. Submit the form: During manual operation, the user will click the submit button of the form; automated operation simulates the action of clicking the submit button through generated keyboard and mouse commands to ensure that the submission process is consistent with manual operation.

[0124] Through the above process, the process of external devices driving the mouse and keyboard to operate computer software through image inference algorithms and large model algorithms is completely consistent with the processing process of manually operating the keyboard and mouse to use computer software, ensuring the accuracy and naturalness of automated operations.

[0125] The present application embodiment discloses an interactive control device based on large model drive, referring to Figure 2 , the devices include but are not limited to:

[0126] The preliminary operation instruction generating module 200 determines the corresponding target computer, obtains the task data issued by the external device connected to the target computer, generates the preliminary operation instruction corresponding to the keyboard and mouse connected to the target computer based on the task data, and sends the preliminary operation instruction corresponding to the keyboard and mouse to the target computer, and controls the target computer to perform the corresponding preliminary operation based on the preliminary operation instruction corresponding to the keyboard and mouse;

[0127] The re-operation instruction generating module 210, after performing the preliminary operation, obtains the display signal on the target computer screen, calls the preset image processing unit, controls the image processing unit to analyze the display signal in real time, and extracts the corresponding key information, calls the large model set in the external device, determines the corresponding target large model in the large model set based on the key information, inputs the key information into the target large model, and generates the corresponding re-operation instruction of the keyboard and mouse;

[0128] The task judgment module 220 controls the target computer to perform a corresponding re-operation based on the corresponding re-operation instructions of the keyboard and mouse. After performing the re-operation, a re-display signal on the target computer screen is obtained, and the re-display signal is used to determine whether all tasks corresponding to the task data have been completed. If not, the image processing unit is called to continue real-time analysis of the re-display signal, and the target computer is controlled to perform a corresponding operation based on the corresponding operation instructions of the keyboard and mouse until all tasks corresponding to the task data are completed.

[0129] Furthermore, the device includes but is not limited to:

[0130] A status result acquisition module, which is used to acquire the running status and operation result corresponding to the target computer based on the operation performed by the target computer;

[0131] The detection module acquisition module inputs the operation status and operation results into a preset anomaly detection model for identification, so as to obtain the corresponding detection results;

[0132] The exception record generation module automatically suspends the current automated process if there is an exception in the detection result, determines the corresponding exception severity level, and uses it to generate the corresponding exception record, where the exception record includes the time, type, cause and related operation log of the exception.

[0133] Furthermore, the device includes but is not limited to:

[0134] The feedback data collection module regularly obtains the user's historical operation records to collect the user's corresponding feedback data, where the historical operation records include operation steps, operation time, and operation results, and the feedback data includes the user's satisfaction evaluation of the operation results, suggested areas for improvement, and problems encountered;

[0135] The operation recognition result recognition module integrates historical operation records and user feedback data to form a corresponding comprehensive data set, and analyzes the comprehensive data set to identify the operation recognition results in user operations, where the operation recognition results include common patterns, error points, and optimization points;

[0136] The parameter adjustment strategy generation module is used to generate the parameter adjustment strategy corresponding to the target large model based on the operation recognition results. The parameter adjustment strategy includes adjusting the weight and bias of the model, modifying the hyperparameters of the model, and increasing or decreasing the number of layers of the model.

[0137] Furthermore, the device includes but is not limited to:

[0138] A current task description generation module identifies the type of task currently being performed by the user based on the real-time analysis operation, and generates a corresponding current task description based on the task type;

[0139] Matching module. Retrieve the user's historical operation records to match the current task description with the tasks in the historical operation records, and select the best operation guide based on the matching results;

[0140] The recommended operation path generation module is used to generate a corresponding recommended operation path based on the best operation guide, and present the recommended operation path to the user through the user interface, wherein the recommended operation path includes specific steps, operation sequence, and precautions.

[0141] Furthermore, the device includes but is not limited to:

[0142] A current context description generation module identifies the user's current context information based on the real-time analysis operation and is used to generate a corresponding current context description;

[0143] The next operation intention prediction module retrieves the user's historical operation records, matches the current context description with the context in the historical operation records, and predicts the user's corresponding next operation intention based on the matching results;

[0144] The next operation instruction generation module is used to generate the corresponding next operation instruction based on the next operation intention, send the next operation instruction to the user, and provide an option for the user to choose whether to accept. If the user confirms to accept the predicted next operation instruction, the cached next operation instruction will be sent to the target computer for operation.

[0145] Furthermore, the device includes but is not limited to:

[0146] A priority evaluation rule calling module is used to call a predefined priority evaluation rule and assign a task priority to the corresponding task based on the task type and the priority evaluation rule;

[0147] Resource availability assessment module, which monitors the resource usage of the target computer in real time and is used to assess the resource availability of the current system. The resource usage includes CPU usage, memory usage, and disk read and write speed.

[0148] The execution order adjustment module is used to adjust the execution order of all tasks to be executed based on task priority and resource availability.

[0149] An embodiment of the present application further discloses an interactive control method based on large model driving, comprising a processor, wherein a program of any one of the interactive control methods based on large model driving described above is run in the processor.

[0150] The embodiment of the present application also discloses a storage medium storing a program of any one of the above-mentioned interactive control methods based on large model driving.

[0151] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An interactive control method based on large model driving, characterized in that: include: Determine a corresponding target computer, obtain task data issued by an external device connected to the target computer, generate preliminary operation instructions corresponding to a keyboard and mouse connected to the target computer based on the task data, send the preliminary operation instructions corresponding to the keyboard and mouse to the target computer, and control the target computer to perform a corresponding preliminary operation based on the preliminary operation instructions corresponding to the keyboard and mouse; After performing the preliminary operation, the display signal on the target computer screen is obtained, a preset image processing unit is called, the image processing unit is controlled to perform real-time analysis on the display signal, and corresponding key information is extracted, a large model set in an external device is called, a corresponding target large model is determined in the large model set based on the key information, the key information is input into the target large model, and a corresponding re-operation instruction of the keyboard and mouse is generated; The target computer is controlled to perform a corresponding re-operation based on the corresponding re-operation instructions of the keyboard and mouse. After the re-operation is performed, a re-display signal on the screen of the target computer is obtained, and it is determined based on the re-display signal whether all tasks corresponding to the task data have been completed. If not, the image processing unit is called to continue to perform real-time analysis on the re-display signal, and the target computer is controlled to perform a corresponding operation based on the corresponding operation instructions of the keyboard and mouse until all tasks corresponding to the task data are completed.

2. The interactive control method based on large model driving according to claim 1 is characterized in that: In the process of the target computer performing corresponding operations based on the corresponding operation instructions of the keyboard and mouse, including the initial operation and the re-operation, the method further includes: Execute operations on the target computer to obtain the corresponding running status and operation results of the target computer; Inputting the operating status and the operation result into a preset abnormality detection model for identification, and obtaining a corresponding detection result; If there is an abnormality in the detection result, the current automated process is automatically suspended, the corresponding abnormality severity level is determined, and a corresponding abnormality record is generated, where the abnormality record includes the time, type, cause and related operation logs of the abnormality.

3. The interactive control method based on large model driving according to claim 1 is characterized in that: The method also includes: Regularly obtain historical operation records of users and collect corresponding feedback data from users. The historical operation records include operation steps, operation time, and operation results. The feedback data includes the user's evaluation of satisfaction with the operation results, areas for improvement, and problems encountered. The historical operation records and user feedback data are integrated to form a corresponding comprehensive data set, and the comprehensive data set is analyzed to identify the operation recognition results in the user operation, wherein the operation recognition results include common patterns, error points and optimization points; A parameter adjustment strategy corresponding to the target large model is generated based on the operation recognition result. The parameter adjustment strategy includes adjusting the weight and bias of the model, modifying the hyperparameters of the model, and increasing or decreasing the number of layers of the model.

4. The interactive control method based on large model driving according to claim 3 is characterized in that: In the process of controlling the image processing unit to perform real-time analysis on the display signal, the method further includes: Identify the type of task currently being performed by the user based on the real-time analysis operation, and generate a corresponding current task description based on the task type; Retrieving the user's historical operation records, matching the current task description with the tasks in the historical operation records, and selecting the best operation guide according to the matching results; A corresponding recommended operation path is generated based on the optimal operation guide, and the recommended operation path is displayed to the user through a user interface, wherein the recommended operation path includes specific steps, operation sequence, and precautions.

5. The interactive control method based on large model driving according to claim 3 is characterized in that: In the process of controlling the image processing unit to perform real-time analysis on the display signal, the method further includes: Identify the user's current context information based on the real-time analysis operation and generate a corresponding current context description; Retrieving the user's historical operation records, matching the current context description with the context in the historical operation records, and predicting the user's corresponding next operation intention based on the matching result; Based on the next operation intention, a corresponding next operation instruction is generated, the next operation instruction is sent to the user, and an option is provided for the user to choose whether to accept. If the user confirms to accept the predicted next operation instruction, the cached next operation instruction is sent to the target computer for operation.

6. The interactive control method based on large model driving according to claim 4 is characterized in that: After the operation based on real-time analysis identifies the type of task currently being performed by the user, the method further includes: Retrieving a predefined priority evaluation rule, and assigning a task priority to the corresponding task based on the task type and the priority evaluation rule; Monitor the resource usage of the target computer in real time and evaluate the resource availability of the current system, including CPU usage, memory usage, and disk read and write speed; The execution order of all tasks to be executed is adjusted based on the task priorities and the resource availability.

7. An interactive control device based on large model drive, characterized in that: include: A preliminary operation instruction generating module determines a corresponding target computer, obtains task data issued by an external device connected to the target computer, generates preliminary operation instructions corresponding to a keyboard and mouse connected to the target computer based on the task data, and sends the preliminary operation instructions corresponding to the keyboard and mouse to the target computer, controlling the target computer to perform a corresponding preliminary operation based on the preliminary operation instructions corresponding to the keyboard and mouse; A re-operation instruction generating module, after performing the preliminary operation, obtains the display signal on the target computer screen, calls a preset image processing unit, controls the image processing unit to perform real-time analysis on the display signal, and extracts corresponding key information, calls a large model set in an external device, determines a corresponding target large model in the large model set based on the key information, inputs the key information into the target large model, and generates a corresponding re-operation instruction for the keyboard and mouse; The task judgment module controls the target computer to perform a corresponding re-operation based on the corresponding re-operation instructions of the keyboard and mouse. After performing the re-operation, a re-display signal on the screen of the target computer is obtained, and the re-display signal is used to judge whether all tasks corresponding to the task data have been completed. If not, the image processing unit is called to continue to perform real-time analysis on the re-display signal, and the target computer is controlled to perform a corresponding operation based on the corresponding operation instructions of the keyboard and mouse until all tasks corresponding to the task data are completed.

8. The interactive control device based on large model driving according to claim 7 is characterized in that: The device also includes: A status result acquisition module, which is used to acquire the running status and operation result corresponding to the target computer based on the operation performed by the target computer; A detection module acquisition module inputs the operating status and the operation result into a preset abnormality detection model for identification, so as to obtain a corresponding detection result; The exception record generation module automatically suspends the current automated process if there is an exception in the detection result, determines the corresponding exception severity level, and uses it to generate the corresponding exception record, where the exception record includes the time, type, cause and related operation log of the exception.

9. An interactive control method based on large model driving, characterized in that: It comprises a processor, in which runs a program of the interactive control method based on large model driving as described in any one of claims 1 to 6.

10. A storage medium, characterized in that: A program for the interactive control method based on large model driving as described in any one of claims 1 to 6 is stored.

Images