An industrial equipment screen remote control method based on artificial intelligence imitation learning

Abstract

The application discloses an industrial equipment screen remote control method based on artificial intelligence imitation learning, and relates to the technical field of industrial automation control, comprising: obtaining a current screen image and inputting an operation semantic understanding model to generate an operation intention vector representing an operation intention; inputting the vector into an industrial process knowledge graph to perform pre-condition and safety mutual exclusion compliance pre-checking; after the checking, generating an action instruction to drive an execution unit operation by a physical execution strategy model; collecting a next frame image after the operation, and calculating a state difference degree between an actual state and an expected state according to an expected state transition model in the knowledge graph; if the difference degree exceeds a threshold, encoding the deviation into a constraint vector to feed back to the operation semantic understanding model to generate a correction intention until the operation is successful. Through the combination of semantic understanding, logical checking and closed-loop feedback, the reliability and safety of remote control are improved.

Description

Technical Field

[0001] This invention relates to the field of industrial automation control technology, specifically to a method for remotely controlling the screen of industrial equipment based on artificial intelligence imitation learning. Background Technology

[0002] In scenarios such as industrial remote operation and maintenance and high-risk inspection, the demand for remote control of equipment screens is becoming increasingly prominent. Traditional methods rely on operators viewing the screen in real time via remote desktop protocols and manually mapping touch coordinates to complete operations. This approach is highly dependent on continuous human intervention, resulting in low efficiency, large response delays, and susceptibility to errors due to fatigue. To reduce reliance on manual intervention, the industry has introduced a behavior cloning-based imitation learning solution. This involves capturing image sequences of expert screen operations using camera devices, simultaneously recording touch actions such as clicks and swipes, and constructing a state-action demonstration dataset. Then, models such as convolutional neural networks or visual Transformers are used to learn the direct mapping from screen images to operational actions. This allows for the automatic generation of commands at the remote end based on the current screen image, driving mechanical units or software to simulate touch control.

[0003] However, this purely visual behavior cloning method exposes deep-seated reliability flaws in real-world industrial screen operation. Industrial screen interfaces often undergo unpredictable changes due to dynamic elements such as process parameter over-limit alarms, safety interlock pop-ups, and permission verification. Existing strategies heavily rely on the distribution and coverage of demonstration data. When faced with pop-up locations, icon distortions, or interface redesigns not seen in the training set, the model is prone to out-of-domain misjudgments, misidentifying inoperable alarm areas as clickable areas, or incorrectly generalizing interactive gestures to critical safety controls. In industrial scenarios, a single misclick or swipe can trigger accidental equipment shutdown, process parameter over-limits, or even safety accidents. A deeper problem is that such methods simplify the control process to a mapping from isolated image frames to actions, completely ignoring the inherent temporal and causal constraints and industrial task logic within the operation sequence. The model cannot understand preconditions such as "interlocks must be released before startup," nor can it perceive the intentional rationality of the current action within the task context. Therefore, when encountering common visual disturbances in industrial settings such as screen aging and dead pixels, micro-vibrations in viewing angle, and gradual changes in lighting, the robustness of imitation learning strategies significantly decreases, frequently resulting in disordered actions, repetitive operations, and interruptions in intent. These issues, through chain reactions, lead to the accumulation of errors in the control sequence, ultimately causing the task to deviate from the safe operating trajectory. Although some studies have attempted to introduce recurrent network modeling of the temporal sequence or employ domain randomization to enhance visual generalization, these methods still fail to address the fundamental deficiency of imitation learning in lacking understanding of operational semantics and process logic, and thus cannot meet the stringent requirements of high reliability and strong safety constraints for industrial remote control. Summary of the Invention

[0004] The purpose of this invention is to provide a method for remotely controlling industrial equipment screens based on artificial intelligence imitation learning, in order to solve the problems mentioned in the background art. To solve the above technical problems, this invention provides the following technical solution: a method for remotely controlling industrial equipment screens based on artificial intelligence imitation learning, comprising:

[0005] Acquire the current image of the industrial equipment screen;

[0006] The current image is input into a pre-trained operation semantic understanding model to generate an operation intent vector that represents the predicted operation intention. The operation intent vector includes the predicted next operation semantics, the functional attributes of the target interface elements, and the location information of the predicted target operation area.

[0007] The operation intent vector is input into a preset industrial process knowledge graph for compliance pre-verification; the industrial process knowledge graph defines operation precondition rules and security mutual exclusion rules.

[0008] When the operation intent vector passes the compliance pre-verification, a physical action instruction is generated by a pre-trained physical execution strategy model based on the operation intent vector and the current image to drive the execution unit to operate the industrial equipment screen.

[0009] After the operation is executed, the next frame image of the industrial equipment screen is obtained;

[0010] Based on the expected state transition model defined in the industrial process knowledge graph, the actual state of the next frame image is compared with the expected state after executing the physical action command to obtain the state difference degree.

[0011] When the state difference exceeds the preset state transition success threshold, the deviation information between the actual state and the expected state is used as a constraint vector and input back into the operation semantic understanding model to generate a corrected operation intent vector. The compliance pre-verification and subsequent steps are then repeated until the state difference meets the state transition success threshold.

[0012] Preferably, the step of inputting the operation intent vector into a preset industrial process knowledge graph for compliance pre-verification specifically includes:

[0013] The operation intent vector is parsed to obtain the predicted semantics of the next operation and the functional attributes of the target interface elements;

[0014] Query the industrial process knowledge graph to determine whether the semantics of the next operation under the current task step node satisfies all defined operation precondition rules;

[0015] If not, then the operation intent vector is blocked, and a modified operation intent vector is generated to satisfy the operation precondition rules.

[0016] Preferably, the step of inputting the operation intent vector into a preset industrial process knowledge graph for compliance pre-verification specifically includes:

[0017] The predicted target operation area is obtained by parsing the operation intent vector.

[0018] Query the industrial process knowledge graph to determine whether the functional attributes of the interface elements located in the target operation area in the current image are defined as prohibited operation elements in the current task state by the security mutual exclusion rule.

[0019] If so, the operation intent vector is blocked, and a modified operation intent vector is generated to circumvent the prohibited operation element.

[0020] Preferably, the training process of the operational semantic understanding model includes:

[0021] Construct a semantic domain dataset, which includes historical images of multiple frames of industrial equipment screens, as well as operation semantic labels, task step nodes, and interface element functional attributes labeled for each historical image corresponding to the operation by experts.

[0022] Using the aforementioned semantic domain dataset, a neural network model is trained that takes screen images as input and operation intention vectors as output, thus obtaining the operation semantic understanding model.

[0023] Preferably, the training process of the physical execution strategy model includes:

[0024] Construct an execution domain dataset, which includes historical images of multiple frames of industrial equipment screens, as well as physical touch action sequences corresponding to each historical image, synchronously recorded by experts during operation;

[0025] The historical image and the operation intent vector obtained from the operation semantic understanding model are used as combined inputs, that is, the visual features of the historical image and the operation intent vector are concatenated and fused and then input into the model. The physical touch action sequence is used as the output target to train another neural network model to obtain the physical execution strategy model.

[0026] Preferably, the expected state transition model defines the expected interface identifier or expected interface layout structure that the industrial equipment screen should display after performing a specific operation in a specified interface state.

[0027] Preferably, the operation prerequisite rule defines the prerequisite operation actions that have been completed or the device state parameters that have been satisfied before the operation action of a specific task is performed.

[0028] Preferably, the security mutual exclusion rule defines that when any dynamic element such as an alarm prompt, a security interlock pop-up, or a permission verification interface appears on the industrial equipment screen, write operations targeting core process parameters or equipment start / stop will be set as prohibited operations.

[0029] Preferably, the execution unit is a robotic arm, a stylus, or a software agent running within the operating system of the industrial equipment.

[0030] Preferably, the industrial equipment is an injection molding machine, a CNC machine tool, a chemical reactor, or a power dispatch control console.

[0031] This invention provides a method for remotely controlling industrial equipment screens based on artificial intelligence imitation learning. It has the following beneficial effects:

[0032] By constructing a technical architecture that integrates an operational semantic understanding model, a physical execution strategy model, and an industrial process knowledge graph, the reliability deficiencies of existing imitation learning solutions caused by a lack of understanding of operational semantics and task causal logic are addressed. The operational precondition rules defined in the industrial process knowledge graph ensure that the system verifies the completion of all prerequisite actions or the fulfillment of equipment status requirements before executing an operation, thus avoiding equipment startup sequence errors caused by skipping necessary steps. Security mutual exclusion rules dynamically bind dynamic elements such as alarm prompts, security interlock pop-ups, and permission verification interfaces to write operations targeting core process parameters or equipment start / stop. Upon detecting such security events, dangerous operational intentions are promptly blocked, preventing security risks arising from misidentifying inoperable areas as operable areas. A closed-loop verification mechanism verifies the effectiveness of each operation by comparing the actual state of the screen after the operation with the expected state defined by the expected state transition model. When the state deviation exceeds a threshold, the deviation information is fed back to the operational semantic understanding model to generate corrective intentions. This mechanism enables the system to self-correct based on the expected state at the task level when faced with visual disturbances, minor changes in the interface, or interface styles that have not appeared in the training data, thus maintaining the correct progression of the control sequence and avoiding action disorder and cumulative errors caused by misjudgment of a single frame.

[0033] By reconstructing the control process into a closed-loop flow of semantic understanding, logic verification, action execution, and result verification, the operation semantic understanding model extracts operation intent vectors rich in task logic information from screen images, rather than simple touch coordinates. This provides interactive semantic input for subsequent logic verification of the industrial process knowledge graph. The physical execution strategy model, after receiving the verified operation intent vector as a constraint, generates precise physical action commands, ensuring that the action output conforms to both the accuracy of visual positioning and the constraints of task logic. This two-way closed-loop design, which uses the causal logic and safety boundaries of the operation task as intrinsic constraints for imitation learning and uses the operation results to correct the operation intent, endows the remote control system with the ability to internalize the industrial task process and perceive the consequences of operations. This improves the reliability and safety of remote control of industrial equipment screens, enabling it to meet the needs of industrial field applications with high requirements for operational accuracy and safety protection. Attached Figure Description

[0034] Figure 1 This is a data flow diagram of the remote control method for industrial equipment screens based on artificial intelligence imitation learning according to the present invention;

[0035] Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Please see Figure 1 and Figure 2 This invention provides a method for remotely controlling industrial equipment screens based on artificial intelligence imitation learning. In implementation, a dual-domain model architecture consisting of an operation semantic understanding model and a physical execution strategy model is first established, and an industrial process knowledge graph for specific industrial tasks is pre-constructed. This industrial process knowledge graph is a structured description of the control task, explicitly defining operation precondition rules, safety mutual exclusion rules, and expected state transition models.

[0038] The industrial process knowledge graph uses an attribute graph model for data storage and representation. Nodes represent task steps, interface elements, or equipment status parameters; edges represent logical relationships between nodes, such as prerequisites, mutual exclusion constraints, or state transitions; attributes store specific rule thresholds, conditional expressions, and action definitions. Specifically, operation prerequisite rules are defined as "prerequisite edges" starting from the 'target action node'. A "prerequisite edge" indicates that the target action logically requires the completion of a prerequisite action; this edge points to the pre-completed 'prerequisite action node' and includes a logical expression of the equipment status parameter as an edge attribute. Safety mutual exclusion rules are defined as "mutually exclusive edges" starting from the 'dangerous action node'. A "mutually exclusive edge" indicates that when an event node triggering a protection state is active, the operation corresponding to the dangerous action node connected to this edge cannot be executed; this edge connects to the 'alarm or interlock event node' triggering the protection state. The expected state transition model is defined as a set of transition paths pointing to the 'expected interface state node', using a combined index of the 'current interface state node' and the 'executed action edge'.

[0039] The precondition rules describe the prerequisite operations or equipment states that have been completed or satisfied before performing a specific operation at a given task step node. The safety mutual exclusion rules define that when dynamic elements such as alarm prompts, safety interlock pop-ups, or permission verification interfaces appear on the screen, operations modifying core process parameters or controlling equipment start / stop are marked as prohibited operations. The expected state transition model describes the expected interface identifier or layout structure that should be displayed on the screen after performing a specific operation in a given interface state. These rules and models together form the foundation for the system to understand the logic of the control task and safety constraints.

[0040] After the remote control task is initiated, a camera device acquires a real-time image of the industrial equipment screen. This image is then input into a pre-trained operation semantic understanding model. Instead of directly outputting touch coordinates, the model outputs an operation intent vector. This vector contains the semantics of the next operation predicted by the model based on the current screen state, the functional attributes of the target interface element to which the operation is directed, and the predicted location information of the target operation area. The operation intent vector is encoded as a fixed-dimensional floating-point vector. The first dimension uses categorical encoding to represent the operation semantic category; the second dimension uses normalized coordinate values ​​to represent the center position and range of the target operation area; and the third dimension uses multi-label encoding to represent the functional attribute category of the target interface element. All these dimensions together form a complete operation intent vector, which is used for parsing in subsequent compliance pre-verification steps. After obtaining the operation intent vector, the system does not immediately convert it into an action but first inputs it into an industrial process knowledge graph for compliance pre-verification. The verification process involves parsing the operation intent vector to obtain the predicted next operation semantics and the functional attributes of the target element. Then, it queries the operation precondition rules in the industrial process knowledge graph to determine whether executing the operation semantics under the current task step node satisfies all preconditions. If not, for example, if the necessary safety interlock release operation was not completed before the current step, the system blocks the operation intent vector and generates a corrected operation intent vector based on the industrial process knowledge graph to satisfy the preconditions, such as generating an operation intent pointing to the interlock release control. Simultaneously, the system also queries safety mutual exclusion rules to determine whether the functional attributes of interface elements located within the predicted operation area on the current screen are defined as prohibited operation elements in the current task state. If so, for example, if the operation area is a parameter input box, but the screen also has an unconfirmed process parameter over-limit alarm icon, the system also blocks the operation intent vector and generates a corrected operation intent vector to avoid the prohibited operation element, such as generating an operation intent pointing to the alarm confirmation button.

[0041] Only after the operation intent vector passes all compliance pre-validations is it, along with the current screen image, passed as input to the physical execution strategy model. This model accepts strong constraints from the operation intent vector, generating precise physical action instructions that specify the action type, precise coordinates, duration, and action sequence. Subsequently, the system drives the aforementioned robotic arm, stylus, or software agent execution units to operate the industrial equipment screen according to these instructions.

[0042] After the operation is completed, the system immediately acquires the next frame image from the industrial equipment screen and initiates a closed-loop verification mechanism. Specifically, based on the expected state transition model in the industrial process knowledge graph, the system determines the expected interface state that should be achieved under the previous interface state and the executed operation, and compares it with the actual interface state reflected in the currently acquired next frame image to calculate the state difference. This state difference reflects whether the operation has achieved the expected task progress effect.

[0043] The comparison between the actual state and the expected state is achieved through feature vector comparison. Specifically, the actual state and the expected state of the next frame image are mapped to fixed-length state feature vectors, and then the difference between these two feature vectors is calculated. The encoder used for mapping directly reuses the feature extraction part before the fully connected layer in the semantic understanding model, outputting a fixed-dimensional feature vector after pooling, without requiring separate training. Before system deployment, the feature vector of the expected state is pre-calculated and stored by inputting the standard template image corresponding to the expected interface state into the same encoder for subsequent real-time comparison. The standard template image is manually captured before system deployment. Specifically, when the industrial equipment is in normal operation, clear screenshots of the corresponding expected interface states are captured from the screen according to the expected interface states defined in the expected state transition model, serving as the standard reference image for that interface state.

[0044] When the state difference exceeds the preset state transition success threshold, for example, if the expected secondary menu does not appear after a click, or the alarm pop-up is not confirmed to disappear, the operation is deemed to have failed to meet expectations. The specific value of the state transition success threshold is determined through testing before system deployment: a series of standard operations are performed in a simulation environment, and the fluctuation range of state difference under normal operation is statistically analyzed. The upper limit of this range is used as the threshold benchmark, and then fine-tuned according to the specific industrial scenario's requirements for false alarm sensitivity. At this time, the system subtracts the feature vector corresponding to the actual state from the feature vector corresponding to the expected state element by element, and the resulting difference vector is used as the constraint vector. The values ​​of each dimension of this constraint vector quantify the direction and degree of deviation between the actual state and the expected state in each visual or semantic dimension. After inputting it into the operation semantic understanding model, it can guide the model to generate a corrected operation intent vector aimed at eliminating the deviation in the next iteration. This constraint vector, as the correction context, is fed back into the operation semantic understanding model. Based on the failed interaction experience and deviation information, the operation semantic understanding model re-understands the current screen state semantically, and then generates a new, corrected operation intent vector. This new operation intent vector will again undergo compliance pre-verification and physical execution processes, forming a closed loop of "understanding, verification, execution, feedback, and re-understanding" until the difference between the screen state obtained after an operation and the expected state falls within the threshold for successful state transition. Only then will the system complete the current step and move on to the next task node.

[0045] The aforementioned closed-loop iteration and verification mechanism endows the control system with the ability to perceive the consequences of operations and self-correct based on task context. When faced with visual disturbances common in industrial settings, such as slight screen aging, minor viewing angle vibrations, or gradual changes in ambient light, even if the physical execution strategy model experiences slight coordinate shifts due to visual noise, or the operation semantic understanding model generates uncertain intentions due to minor interface changes, the subsequent state verification stage can promptly identify the fact that the operation has not met expectations by comparing it with the expected state at the task level, and correct the subsequent intention generation process through feedback deviation information. This process eliminates the absolute dependence on the accuracy of single-frame image recognition, ensuring the correctness of the control sequence from a macroscopic level of task flow. At the same time, this mechanism also enables the system to safely interact with the environment when faced with pop-up styles or interface layouts that have not appeared in the training data, based on the feedback of "expected state not achieved," by attempting correction rather than blindly executing, thereby avoiding the risk of misidentifying inoperable areas as clickable areas due to misjudgment from outside the domain, or executing incorrect gestures on critical safety controls. The entire control process remains within the logical and safety boundaries defined by the industrial process knowledge graph, avoiding problems such as disordered control sequences, interrupted intent, and accumulated errors caused by a lack of understanding of operational semantics and task causal logic.

[0046] After the operation semantic understanding model generates an operation intent vector based on the current screen image, the vector is first parsed to extract the predicted next operation semantics contained within it, as well as the identifier and functional attributes of the target interface element to which the operation semantics points. For example, the parsing result may indicate that the operation semantics is "numerical writing" or "control activation", and explicitly point out that its target is a "parameter input box" or "start button" on the interface.

[0047] Subsequently, the system inputs the operational semantics and the current task's current step node as query conditions into a pre-defined industrial process knowledge graph. This industrial process knowledge graph contains a clear set of operational precondition rules. These rules are not isolated but strictly bound to specific task step nodes. The system performs a query and comparison process to determine whether the extracted operational semantics at the current task step node have satisfied all associated preconditions. These preconditions typically define prior actions completed beforehand or internal equipment state indicators reached. For example, if the industrial process knowledge graph defines a rule that: before performing the "start main pump" operation, the completed prior action is "confirm cooling water valve is open," and the satisfied equipment state is "safety interlock signal is released."

[0048] If, upon querying, it is found that any of the necessary preconditions for the operation are not met in the current system state record or the historical sequence of completed actions, it indicates that the intent generated by the operation semantic understanding model, while reasonable at the screen visual level, is illegal and inappropriately timed at the operational logic level of the industrial task. In this case, the system will not pass the operation intent vector to the subsequent physical execution model, but will instead block the original operation intent vector.

[0049] While blocking the operation intent vector, the system does not simply abandon the operation or issue an alarm. Instead, it proactively corrects the intent based on the industrial process knowledge graph. Specifically, the system uses the unmet precondition rules to reverse-search and locate one or more prerequisite operations required to meet those preconditions. Based on this, the system generates a new operation intent vector in the semantic space, designed to bridge the precondition gap. The semantics of the operation expressed by this corrected vector changes from the original "numerical writing" or "starting the equipment" to "opening the cooling water valve" or "confirming the release of the safety interlock." This new operation intent vector then replaces the original blocked vector and re-enters the compliance pre-verification process. After its own preconditions are met, it is finally transformed into a physical action command. Through this mechanism, the system's behavioral output is forcibly constrained within a causal chain that conforms to the logic of the industrial process, preventing safety issues such as incorrect equipment startup sequence or process disorder caused by skipping necessary steps.

[0050] Once the operation intent vector generated by the operation semantic understanding model passes the precondition verification, the system will parse the vector and extract the predicted target operation area information. This target operation area is not an abstract instruction, but rather includes the coordinate range or positioning bounding box of the interaction that the operation intent intends to exert on the two-dimensional space of the screen, as well as the identifier of the candidate interface element corresponding to the area.

[0051] Next, the system uses the current task status and the target operation area as joint query conditions to access the set of safety mutual exclusion rules in the industrial process knowledge graph. These safety mutual exclusion rules do not statically mark certain fixed coordinate areas on the screen as permanently prohibited zones; instead, they define the association between the functional attributes of interface elements and the dynamic states during task execution. For example, a rule could be set such that clicking on parameter setting input boxes is allowed when the industrial equipment is in normal operation; however, if the system detects an alarm icon indicating that a process parameter has exceeded its limit at any location on the screen, and the alarm is in an unconfirmed state, the rule defines "parameter writing" operations as prohibited. Alternatively, a rule could be set such that, in equipment maintenance mode, the software-mapped area of ​​the "emergency stop" physical button on the screen is defined as prohibiting clicks by the remote imitation learning system.

[0052] During system verification, the system identifies interface elements located within the predicted target operation area in the current screen image and determines the functional attributes assigned to these elements according to rules in the current state. The system compares these functional attributes with the set of attributes defined as "prohibited operation" in the security mutual exclusion rules. If a match is found—for example, if the predicted target operation area happens to be located on the "Cancel" button of a recently popped-up security interlock confirmation dialog box, and this button is defined as a prohibited operation element in the current interlock state—it indicates that although the current operation intent may be visually similar to the training samples, its target is dangerous or violates regulations in the current state.

[0053] Once such a match is detected, the system immediately blocks the transmission of the operation intent vector, preventing it from reaching the physical execution strategy model. Simultaneously, the system analyzes the type of the prohibited operation element and its associated safety event, retrieving standard operating procedures from the industrial process knowledge graph to deactivate the safety interlock or properly handle the current alarm. Based on this procedure, the system generates a modified operation intent vector to circumvent the prohibited operation element. For example, the modified intent is no longer clicking the "Cancel" button, but instead points to the "Confirm Alarm" or "Initiate Safety Reset" control. This modified operation intent vector replaces the original dangerous intent and re-enters the compliance pre-verification process, ensuring that the system's operational behavior remains within the limits allowed by the safety state machine whenever any dynamically changing safety prompts appear on the industrial screen.

[0054] The foundation of this model is the establishment of a semantic domain dataset. The acquisition process for this dataset differs fundamentally from traditional behavioral cloning methods that only record screen images and touch coordinates. During the process of an expert operating an industrial equipment screen to complete a standard work task, a camera simultaneously captures a sequence of screen images. For each frame in the sequence, a multi-dimensional semantic label set is assigned, either by the system or through manual annotation. This label set contains at least three levels of information. The first level is the operational semantic label, which explicitly describes the essential meaning of the operation performed by the expert at that frame. For example, this label could be "system menu navigation," "process parameter selection," "numerical increment adjustment," "valve switch," or "alarm information confirmation," rather than a simple physical description like "click at coordinate X." The second level is the current task step node, which associates the time of the image frame with a specific logical stage in the entire work process, such as associating it with the "process three pressure parameter setting stage" or the "pre-start safety interlock status check stage." The third layer consists of the functional attributes of active elements in the interface. These attributes define the interactive controls or key status icons displayed on the current screen. For example, an input box area can be labeled as an "operable numerical writing area", an alarm indicator icon can be labeled as an "inoperable alarm status indicator", and a button can be labeled as an "operable page navigation control".

[0055] After constructing the semantic domain dataset, a neural network model was trained under supervision. The input to this neural network model was a single-frame historical image of an industrial equipment screen, and its output was an operation intent vector corresponding to the input image, composed of the encodings of the aforementioned operation semantic labels, task step nodes, and interface element functional attributes. During training, the operation intent vector output by the model was compared with the labeled real operation intent vector. The difference between the two was used as the loss signal, and the model parameters were iteratively optimized using the backpropagation algorithm until the model's prediction accuracy on the validation set met the preset requirements. During training, the model learned how to identify the current task stage, determine which areas are currently operable controls and their corresponding functions from the pixel features of the screen image, and infer the operation semantics that should be executed to advance the task. The operation semantic understanding model obtained through this training process does not output a specific physical coordinate, but rather transforms the screen visual state into an intent expression rich in task logic information and operation semantic constraints. This provides a structured decision-making basis that can be queried and reasoned using an industrial process knowledge graph for subsequent compliance pre-verification steps.

[0056] Training this physical execution strategy model first requires constructing an execution domain dataset. This dataset differs from the aforementioned semantic domain dataset, focusing instead on the physical execution level of expert operations. During expert manipulation of industrial equipment screens, cameras also capture screen image sequences, but the core information recorded synchronously is the expert's physical touch action sequence. This sequence is not merely isolated two-dimensional coordinates, but rather structured data containing action type, action coordinates, action duration, and temporal intervals strictly aligned with each screen frame. Action type clearly distinguishes single clicks, consecutive double clicks, long presses, single-finger swipes, and multi-finger gestures. Action coordinates record the precise position of the touch center point in the screen coordinate system. Action duration records the time span from pressing to releasing the touch action. This refined execution-level data constitutes the supervisory signal for the model to learn "how to operate."

[0057] During the model training phase, the input to the neural network model is not merely historical screen images. It is a combined input consisting of two parts. The first part is the screen image itself, corresponding to a specific historical image frame, from the execution domain dataset. The second part is the operation intent vector, obtained from the pre-trained operation semantic understanding model, corresponding to the same historical image frame. This operation intent vector acts as an intent condition and logical boundary constraint, informing the physical execution strategy model what semantic-level operation should be performed in the current screen state, and what functional attributes the target element of that operation should possess.

[0058] This neural network model fuses the two inputs: first, it extracts visual feature vectors and operation intent vectors from historical images separately through a feature extraction network; then, it concatenates the two vectors in a high-dimensional feature space to form a joint feature representation. The feature extraction network shares the underlying convolutional parameters with the operation semantic understanding model. The concatenated joint feature representation is input to a regression module consisting of a multi-layer fully connected network. This module takes the joint feature representation as input and outputs the components of the physical touch action sequence. During training, expert physical touch action sequences recorded in the execution domain dataset are used as the supervision target. Backpropagation is used to optimize the model parameters, minimizing the error between the output action sequence and the expert-annotated sequence. Through this training method, the model learns a conditional mapping relationship from "visual state and operation intent" to "physical action." The physical execution strategy model obtained after training convergence can generate a set of physical action instructions containing action type, accurate coordinates, and operation duration when it receives a new screen image and a validated operation intent vector, thereby driving the execution unit to accurately complete the screen interaction operation that matches the intent.

[0059] This expected state transition model is a component of the industrial process knowledge graph. It uses step nodes in the operation task as indexes and defines the corresponding expected result for each possible operation interaction. Specifically, the model stores a series of mapping relationships, each of which relates to three elements: the current interface state identifier, the type of operation action performed by the system in this state, and the expected interface identifier or expected interface layout structure that should be displayed on the screen after the operation is performed.

[0060] Interface state identifiers are abstract names for a type of screen display at the task logic level. They can be defined by specific window titles, specific combinations of interface elements, or specific sets of status indicator icons appearing on the screen. For example, an interface state can be identified as "Main Operation Monitoring Interface" or "Process Parameter Setting Interface." Expected interface identifiers, on the other hand, are identifiers indicating the next interface state the system should switch to after performing a specific operation in the previous state. For example, if the current interface state is "Main Operation Monitoring Interface," and the operation "Activate System Menu" is performed, the expected interface identifier afterward is defined as "Main Menu Expand Interface."

[0061] For operations that involve only partial changes and do not involve a full page transition, the expected state transition model defines the correct state after the operation by describing the expected interface layout structure. This description focuses on the distribution of key functional areas in the interface and the attribute changes of specific controls. The expected interface layout structure is described using a hierarchical data structure, including the following elements: the type of interface element, the normalized position coordinates of each element on the screen, and the key state attributes of each element. During state comparison, the system extracts the corresponding attributes of the actual interface elements from the next frame image and compares them item by item with the attributes of each element defined in the expected interface layout structure, summarizing the state difference. For example, in the "Alarm Processing Interface," after performing a "single click" operation on the "Alarm Confirmation" button, the model does not expect a completely new interface identifier, but defines the expected changes to the current interface layout structure: the alarm pop-up window originally located in the center of the screen should disappear, and the alarm status indicator bar at the top of the screen should change from a flashing red state to a solid green state. For example, in the "Parameter Fine-tuning Interface", after performing the "Slide Up" operation on the numerical adjustment control, the model defines the expected layout structure change of the target numerical display area, that is, the numerical display content in the area should change accordingly according to the sliding direction.

[0062] When the closed-loop verification mechanism is activated, the system retrieves the corresponding expected interface identifier or expected interface layout structure from the expected state transition model based on the interface state identifier recorded before the current operation and the type of operation performed. Subsequently, the system analyzes the next frame of the screen image acquired after the operation, extracts the actual interface identifier and layout structure features, and compares the two to obtain the degree of difference used to determine whether the operation has achieved the expected state. In this way, the system's evaluation of the operation result no longer rests on the visual surface similarity but delves into the level of interface functional state and task flow, providing a clear and interpretable decision-making basis for whether to trigger closed-loop iterative correction.

[0063] In the industrial process knowledge graph, precondition rules are specifically defined as a data structure strongly associated with task step nodes. Each rule explicitly specifies one or more prerequisite operations or one or more equipment state parameter conditions that have been completed or met before a specific operation is to be performed at a particular task step node.

[0064] For prerequisite rules that are conditional on specific operations, they are defined by binding the target operation to a list of prerequisite operations. For example, in the operation flow of a certain type of injection molding machine, under the task step node of "mold closing control," the rule can be defined as follows: Execute the target operation "start mold closing action," whose prerequisite operation list includes the operation "confirm safety door is closed," and the completion status of this operation has been recorded by the system. When the system performs compliance pre-verification, it will retrieve the historical sequence of completed operations. Only when all prerequisite operations in the list have been successfully executed and marked as completed is the prerequisite considered satisfied.

[0065] For precondition rules based on equipment status parameters, the definition method involves binding the target operation to threshold conditions of one or more internal equipment status bits or analog quantities. These equipment status parameters are not obtained directly from screen images but through data interfaces with industrial equipment controllers, reflecting the actual physical or logical state of the equipment. For example, in the operation flow of a certain type of CNC machine tool, under the task step node of "spindle start," the rule can be defined as follows: to execute the target operation of "spindle forward rotation start," the equipment status parameter conditions that must be met are "lubrication system oil pressure status bit is normal," "spindle coolant flow status bit is non-zero," and "the analog value of the current spindle speed parameter meets the definition range of the static state." During the system's verification, these internal equipment status parameters are queried in real time and compared with the preset thresholds or status bits in the rule. Only when all parameter conditions are met is the precondition considered passed.

[0066] If the system finds any unmet precondition during the verification process, it determines that the intent generated by the current operation semantic understanding model is inappropriately timed in the causal chain of the task logic. The system then blocks the operation intent vector and, based on the rules of those unmet preconditions, generates a corrected operation intent vector designed to prioritize bridging these precondition gaps. This mechanism ensures that the control sequence strictly follows the inherent causal logic of the industrial task at all times, preventing equipment damage or safety accidents caused by skipping necessary preparation steps.

[0067] The definition of security mutual exclusion rules in the industrial process knowledge graph is not to statically mark certain fixed coordinate areas on the screen as permanent restricted areas, but to establish a dynamic operation permission constraint mechanism triggered by the screen interface state. This rule mutually excludes specific screen visual events with specific operation action categories, and its defined structure includes two parts: trigger conditions and a set of prohibited operations.

[0068] The triggering conditions section defines the screen state characteristics that require activation of safety mutual exclusion protection. These characteristics specifically point to three types of dynamic elements that may appear on the industrial equipment screen during operation. The first type is alarm prompts, which can manifest as alarm icons popping up anywhere on the screen, alarm text bars, flashing red parameter values, or a dedicated alarm information list. The second type is safety interlock pop-ups, which are usually modal dialog boxes that automatically pop up when the equipment detects abnormal opening of a safety door, pressing of an emergency stop button, or malfunction of a protective device, requiring the operator to confirm or perform a reset operation. The third type is the access verification interface, which is activated when modifying core process formulas, entering equipment maintenance mode, or performing dangerous actions, requiring the input of an operation password or secondary authentication of an access token.

[0069] The prohibited operation action set defines the categories of operation actions that the system prohibits from executing when any of the above trigger conditions are active on the screen. These prohibited operation actions do not broadly refer to all screen interactions, but are precisely limited to write operations targeting core process parameters or equipment start / stop. Write operations targeting core process parameters specifically cover numerical modification commands, value increment / decrement commands, or parameter saving commands for key process variables such as temperature setpoints, pressure setpoints, speed setpoints, flow setpoints, position coordinates, and recipe data. Write operations targeting equipment start / stop specifically cover start commands, stop commands, reset commands, and mode switching commands sent to the equipment controller.

[0070] During system operation, the security mutual exclusion verification mechanism continuously parses the current screen image and the device's operating context. Once any dynamic element in the current screen, such as an alarm prompt, security interlock pop-up, or permission verification interface, is detected, the rule is activated. At this point, the system compares the target operation semantics extracted from the operation intent vector generated by the operation semantic understanding model with the set of prohibited operations defined in the rule. If the operation semantics belongs to the write operation category targeting core process parameters or device start / stop, the system will immediately block further transmission of the operation intent vector, regardless of whether the area indicated by the operation intent appears operable at the screen visual level. This ensures that any remote control behavior that could alter the core operating state or core formula of the device cannot be executed when the device is in an abnormal state such as an abnormal alarm, security interlock, or pending permission verification.

[0071] When the industrial equipment screen is a physical touchscreen and the corresponding hardware is already deployed on-site, a robotic arm can be used as the execution unit. The robotic arm's end effector is equipped with a conductive touch head to simulate human touch operation. The robotic arm's control system receives physical motion commands output from the physical execution strategy model. These commands include the motion type, the target coordinates in the screen coordinate system, the motion duration, and the timing interval between motions. The robotic arm controller parses these commands into motion control signals for each joint motor, driving the end effector to contact or glide across the target coordinate position on the screen surface with the specified motion type and maintain the position for the specified duration, thus completing one screen interaction action. After the operation is executed, the system captures subsequent images of the screen using a camera device to support the operation of the closed-loop verification mechanism.

[0072] In applications with limited space or requiring higher precision, an end effector in the form of a stylus can be used as the execution unit. This stylus is driven by a high-precision motion platform, and its control method is similar to that of a robotic arm, receiving physical motion commands and converting them into motion control signals. When the stylus performs clicks, swipes, or long presses, the contact area and pressure between the stylus and the screen can be controlled by the motion platform to adapt to different types of touchscreen technologies.

[0073] When the object of remote control is a software interface running on the operating system of industrial equipment, and the remote control system has obtained the permission to inject events within the target operating system, the execution unit can be a software agent running within that operating system. This software agent resides in the industrial equipment's operating system as a process or service, receiving physical action instructions generated by the physical execution strategy model through a network communication interface. The software agent maps the action type and target coordinates specified in the instructions to input events at the operating system's underlying layer. For example, for a click operation at coordinates, the software agent constructs a touchscreen input event containing coordinate parameters and action type parameters, and directly injects it into the operating system's event distribution management module. The operating system then distributes this event to the corresponding application interface control, generating an interface response equivalent to a physical touch operation. This software agent can also simulate input event sequences of complex gestures such as swiping and long-pressing. Regardless of the execution unit form used, its role is to act as the final controlled link in the entire closed-loop control system, faithfully and accurately executing the security-verified physical action instructions output by the intelligent decision-making link, thereby completing the remote control of the industrial equipment screen.

[0074] For injection molding machines, screen control involves precise settings for multiple parameters such as mold closing, injection pressure, holding time, and temperature. Safety mechanisms such as safety door interlocks and emergency stop buttons are commonly used in the operation process. When remotely controlling the injection molding machine screen, if traditional imitation learning methods are used, visual disturbances may cause the safety door status indicator area to be misidentified as an operable control, or attempts to modify injection parameters may be made before the interlock is released, potentially leading to mold damage or personal injury. This method uses pre-condition rules in the industrial process knowledge graph to pre-bind the mold closing action with the safety door closing state. Simultaneously, through safety mutual exclusion rules, it blocks all written operations targeting mold actions and heating control in emergency stop or safety door open states. Furthermore, a closed-loop verification mechanism self-corrects deviations, helping to keep the remote control of the injection molding machine screen within the constraints of safety logic.

[0075] For CNC machine tools, the screen interface is commonly used to set spindle speed, feed rate, tool compensation values, and to start and stop machining programs. In remote control scenarios, it is crucial to ensure that the lubrication and cooling systems are ready before spindle startup, and that tool parameters are not accidentally modified during high-speed cutting. This method's industrial process knowledge graph can set spindle startup and equipment status parameters such as normal lubrication pressure and coolant flow as prerequisites. When the operation semantic understanding model generates a parameter modification intent, a safety mutual exclusion rule will prohibit the writing of process parameters when the machine tool is in automatic machining mode and the tool is rotating, thereby preventing tool collisions or workpiece scrap due to remote misoperation.

[0076] For chemical reactors, screen controls typically involve setting reaction conditions such as temperature, pressure, and stirring rate, as well as controlling the on / off states of feed valves, discharge valves, and heating and cooling cycles. In this scenario, operational errors can lead to major safety accidents such as overheating, overpressure, and leakage of toxic materials. The expected state transition model in this method can be used to describe the correspondence between valve on / off states and screen icon color changes. The closed-loop verification mechanism can confirm whether the operation has truly taken effect by detecting whether the valve icon state meets expectations. Simultaneously, the precondition rules can enforce that "the stirring motor is running" and "the pressure interlock is within the normal range" must be confirmed before executing "start heating," preventing reaction runaway caused by incorrect operating sequence from the outset.

[0077] For power dispatch control consoles, the screen interface displays key information such as the primary wiring diagram of the power grid, circuit breaker status, load curves, and protection circuit breakers. Any remote control operation involving opening / closing or setting modification must adhere to strict switching operation sequences and safety procedures. This method transforms the power grid dispatch operation ticket system into pre-operation condition rules within an industrial process knowledge graph, ensuring that before executing a "disconnect circuit breaker" operation, it is confirmed that "the load-side isolating switch is in the open position." Safety mutual exclusion rules prohibit remote control closing operations when equipment is in a maintenance-suspended state or when relay protection signals have not reset. A closed-loop verification mechanism confirms whether the remote control operation was completed as expected by comparing whether the circuit breaker graphic symbol on the screen changes from the closed to the open position after the operation, providing logical-level security for remote control operations with high reliability requirements, such as power dispatch.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for remotely controlling the screen of industrial equipment based on artificial intelligence imitation learning, characterized in that, Includes the following steps: Step 1: Acquire the current image of the industrial equipment screen; Step 2: Input the current image into a pre-trained operation semantic understanding model to generate an operation intent vector that represents the predicted operation intent; the operation intent vector includes the predicted next operation semantics, the functional attributes of the target interface elements, and the location information of the predicted target operation area; Step 3: Input the operation intent vector into the preset industrial process knowledge graph for compliance pre-verification; The industrial process knowledge graph is defined with operation precondition rules and security mutual exclusion rules. Step 4: When the operation intent vector passes the compliance pre-verification, a physical action instruction is generated by the pre-trained physical execution strategy model based on the operation intent vector and the current image to drive the execution unit to operate the industrial equipment screen. Step 5: After the operation is executed, acquire the next frame image of the industrial equipment screen; Step Six: Based on the expected state transition model defined in the industrial process knowledge graph, compare the actual state of the next frame image with the expected state after executing the physical action command to obtain the state difference degree; Step 7: When the state difference exceeds the preset state transition success threshold, the deviation information between the actual state and the expected state is used as a constraint vector and input back into the operation semantic understanding model to generate a corrected operation intent vector. The compliance pre-verification and subsequent steps are repeated until the state difference meets the state transition success threshold.

2. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The step of inputting the operational intent vector into a preset industrial process knowledge graph for compliance pre-verification specifically includes: The operation intent vector is parsed to obtain the predicted semantics of the next operation and the functional attributes of the target interface elements; Query the industrial process knowledge graph to determine whether the semantics of the next operation under the current task step node satisfies all defined operation precondition rules; If not, then the operation intent vector is blocked, and a modified operation intent vector is generated to satisfy the operation precondition rules.

3. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The step of inputting the operational intent vector into a preset industrial process knowledge graph for compliance pre-verification specifically includes: The predicted target operation area is obtained by parsing the operation intent vector. Query the industrial process knowledge graph to determine whether the functional attributes of the interface elements located in the target operation area in the current image are defined as prohibited operation elements in the current task state by the security mutual exclusion rule. If so, the operation intent vector is blocked, and a modified operation intent vector is generated to circumvent the prohibited operation element.

4. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The training process of the operational semantic understanding model includes: Construct a semantic domain dataset, which includes historical images of multiple frames of industrial equipment screens, as well as operation semantic labels, task step nodes, and interface element functional attributes labeled for each historical image corresponding to the operation by experts. Using the aforementioned semantic domain dataset, a neural network model is trained that takes screen images as input and operation intention vectors as output, thus obtaining the operation semantic understanding model.

5. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The training process of the physical execution strategy model includes: Construct an execution domain dataset, which includes historical images of multiple frames of industrial equipment screens, as well as physical touch action sequences corresponding to each historical image, synchronously recorded by experts during operation; The historical image and the operation intent vector obtained from the operation semantic understanding model are used as combined inputs, which are obtained by concatenating and fusing the visual features of the historical image and the operation intent vector; the physical touch action sequence is used as the output target to train another neural network model to obtain the physical execution strategy model.

6. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The expected state transition model defines the expected interface identifier or expected interface layout structure that the industrial equipment screen should display after performing a specific operation in a specified interface state.

7. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The operation prerequisite rules define the prerequisite operations or device state parameters that have been completed or satisfied before the operation action of a specific task is performed.

8. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The security mutual exclusion rule defines that when any dynamic element appears on the industrial equipment screen, such as an alarm prompt, a security interlock pop-up, or a permission verification interface, write operations on core process parameters or equipment start / stop will be set as prohibited operations.

9. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The execution unit is a robotic arm, a stylus, or a software agent running within the operating system of the industrial equipment.

10. The method for remote control of industrial equipment screens based on artificial intelligence imitation learning according to claim 1, characterized in that, The industrial equipment mentioned includes injection molding machines, CNC machine tools, chemical reaction kettles, or power dispatch control consoles.

Images