Intelligent console control method and system based on multimodal interaction

Abstract

This invention relates to the field of industrial control technology, and more particularly to a control method and system for an intelligent console based on multimodal interaction. The method acquires multimodal input data, extracts modal semantic features and maps them to a unified semantic space to identify control intentions, performs semantic conflict disambiguation based on a knowledge base, decomposes the intention into sub-tasks and solves for state transition paths through backpropagation, establishes a mapping relationship between task execution sequences and device state spaces to identify implicit dependencies, and generates a control command sequence after topology rearrangement, which is then sent to the corresponding devices and feedback is collected. This invention improves the semantic understanding accuracy and task execution efficiency of a multimodal interactive console.

Description

Technical Field

[0001] This invention relates to the field of industrial control technology, and in particular to a method and system for controlling an intelligent console based on multimodal interaction. Background Technology

[0002] In industrial control, traditional control consoles typically rely on a single interaction channel (such as a touch panel or physical buttons) to receive operating commands. Operators must control the equipment through a fixed interface hierarchy or a preset command sequence. With the development of multimodal interaction technology, a few solutions have attempted to integrate multiple input methods such as voice, gestures, and eye tracking. However, the conventional approach is simply to concatenate or weighted sum the features of different modalities without establishing a unified semantic representation system. This shallow integration method cannot resolve the ambiguity of intent caused by information contradictions or vague expressions between modalities (such as inconsistent voice commands and gesture directions), leading to chaotic control logic or misoperation.

[0003] Furthermore, existing control methods generally employ predefined task templates or rule-based sequential arrangement when generating execution sequences, lacking dynamic analysis of state constraints and dependencies in complex tasks. For example, when multiple subtasks involve shared equipment resources or require satisfying preceding and subsequent state conditions, conventional systems cannot automatically identify implicit dependencies and mechanically issue tasks according to the order of instruction input, easily leading to resource contention, state conflicts, or execution stagnation. Simultaneously, due to the lack of real-time correlation and iterative optimization of equipment state feedback, the generated instruction sequences often contain redundant steps, making it difficult to flexibly adjust execution paths in dynamically changing industrial environments, thus reducing control efficiency and system robustness. Summary of the Invention

[0004] The embodiments of the present invention provide a smart console control method and system based on multimodal interaction, which can solve the problems in the prior art.

[0005] A first aspect of the present invention provides a smart console control method based on multimodal interaction, comprising: Acquire multimodal input data from different interaction channels of the industrial console; Modal semantic features are extracted from the input data of each modality, and the semantic features of each modality are mapped to a unified semantic space to obtain fused semantic features and identify control intentions; Based on a pre-built industrial control semantic knowledge base, semantic conflicts in the control intent are detected and disambiguated to obtain the disambiguated intent expression. The intent expression is decomposed into multiple sub-control tasks. The pre-state conditions and post-state changes of each sub-control task are defined. The state transition path from the current state to the target state is solved by backpropagation of the target state constraints. The optimal state transition path is selected and the task execution sequence is generated. A mapping relationship between the task execution sequence and the device state space is established. The implicit dependencies between each sub-control task are identified by solving the state variable constraints in the mapping relationship. The task execution sequence is topologically rearranged and the execution time window and target execution device are allocated to generate a control instruction sequence. The control command sequence is sent to the corresponding device on the industrial control console to drive the device to perform operations and collect device execution status feedback information.

[0006] Modal semantic features are extracted from the input data of each modality, and the semantic features of each modality are mapped to a unified semantic space to obtain fused semantic features and identify control intentions, including: Modality-specific encoding is performed on the input data of each modality to extract semantic features. Semantic decomposition is performed to obtain multiple semantic sub-units. The semantic similarity between semantic sub-units of different modalities is calculated. Based on the semantic similarity, cross-modal semantic sub-unit pairs representing the same semantic content are identified and used as semantic pairs. Spatial alignment constraints are applied to the semantic pairs and mapped to a unified semantic space. Semantic clusters are formed by clustering the semantic sub-units in the unified semantic space based on the spatial distance between them. The weighted average of the semantic sub-units in each semantic cluster is then used to obtain the cluster representation vector. Semantic role labeling is performed on each cluster representation vector to identify the corresponding control object role, control action role, and control parameter role. The cluster representation vectors of different roles are then decoded into control object identifier, control action type, and control parameter value, respectively. A triplet structure is constructed based on the semantic dependency relationship between the control object identifier, the control action type, and the control parameter value. The control object identifier is the subject, the control action type is the predicate, and the control parameter value is the object. The triplet structure is used as the control intent.

[0007] Based on a pre-built industrial control semantic knowledge base, semantic conflicts in the control intent are detected and disambiguated to obtain the disambiguated intent expression, including: Using the control object identifier, control action type, and control parameter value in the control intent as semantic nodes, the semantic dependency relationship between each semantic node is retrieved from the industrial control semantic knowledge base, and directed connection edges are established between each semantic node to construct the intent semantic graph based on the semantic dependency relationship. Traverse the directed connection edges in the intent semantic graph to extract the semantic consistency conditions marked on the edges, substitute the semantic nodes connected by the directed connection edges into the semantic consistency conditions for verification, and mark the directed connection edges as conflicting edges and record the source and target nodes connected by the conflicting edges. For conflict edges, the set of replaceable nodes for the source node and the set of replaceable nodes for the target node are retrieved from the industrial control semantic knowledge base. The set of replaceable nodes is searched for node replacement combinations that meet the semantic consistency condition. The node replacement combination with the smallest sum of semantic distances to the original node is selected for replacement. The replaced nodes are used to update the intent semantic graph and the attribute values ​​of each node are extracted. The extracted attribute values ​​are recombined into a triple structure in the order of control object identifier, control action type and control parameter value. The recombined triple structure is used as the disambiguated intent expression.

[0008] The intent expression is decomposed into multiple sub-control tasks. Preconditions and subsequent state changes are defined for each sub-control task. The state transition path from the current state to the target state is solved through backpropagation of target state constraints. The optimal state transition path is selected, and a task execution sequence is generated, including: Dependency parsing is performed on the intent expression to identify the dominance relationship between the control action type and the control object identifier. Based on the dominance relationship, the control action type is split into the main control action and the auxiliary control action, which are respectively corresponding to sub-control tasks. The current state vector is constructed by reading the device status register of the industrial console and the target state vector is constructed by inferring the expected operating state based on the control parameter values. After projecting the current state vector and the target state vector onto a unified state space, the state coordinate offset caused by each sub-control task in the unified state space is calculated as the subsequent state change. The state coordinates before the offset are used as the pre-state conditions of each sub-control task. Starting from the target state vector, a reverse search is performed along the state gradient direction. Each search selects the sub-control task that makes the state coordinates closer to the current state vector and records it and its corresponding state coordinates as search nodes. When a search node enters the neighborhood of the current state vector, the reverse search terminates. Connect each search node to form a state transition path. Calculate the direction and magnitude changes of adjacent state coordinates in each state transition path to calculate the path smoothness. Select the state transition path with the highest path smoothness as the optimal state transition path. Extract the corresponding sub-control tasks according to the reverse order of the search nodes in the optimal state transition path and arrange them sequentially to generate a task execution sequence.

[0009] A mapping relationship is established between the task execution sequence and the device state space. Implicit dependencies between sub-control tasks are identified by solving the state variable constraints in the mapping relationship. The task execution sequence is topologically rearranged, and execution time windows and target execution devices are allocated to generate a control instruction sequence, including: The state space dimension definition corresponding to each sub-control task in the task execution sequence is retrieved from the device state model library. The control parameter values ​​are projected onto the state space dimension to form state vector coordinates. The modification range of each sub-control task is recorded as the state influence domain. Calculate the intersection of the state influence domains of the preceding and subsequent sub-control tasks and determine the dependencies. Read the boundary safety constraints from the industrial control semantic knowledge base to verify whether the state influence domains have exceeded the limits and mark the sub-control tasks that need to be protected in advance. Construct a task constraint relationship network. Identify the strongly connected components in the task constraint relationship network and merge them into atomic execution units, then rearrange the task execution sequence in ascending order of node hierarchy depth; Obtain historical execution time statistics for each sub-control task after rearrangement and calculate conservative duration estimates based on the statistical data. Sequentially determine the execution time window for each sub-control task in a time cursor progressive manner. Based on the execution time window of each sub-control task, query the device topology diagram to obtain executable device nodes, evaluate the task queue length of each device node and select the shortest one as the target execution device, encode the sub-control task into the target device instruction message and generate a control instruction sequence according to the timing.

[0010] Calculate the intersection of the state influence domains of the preceding and subsequent sub-control tasks and determine their dependencies. Read boundary safety constraints from the industrial control semantic knowledge base to verify whether the state influence domains have exceeded their limits and mark sub-control tasks requiring prior protection. Construct a task constraint relationship network, including: The sub-control tasks in the task execution sequence are paired up, and the boundary coordinates of the state influence domain of the preceding and subsequent sub-control tasks in each task pair are extracted. The overlapping intervals are calculated in each dimension of the state space and combined to form the intersection region of the state influence domain. Determine whether the volume of the intersection region of the state influence domain is zero. If the volume is non-zero, determine that the subsequent sub-control task depends on the preceding sub-control task and record the dependency relationship. Based on the boundary coordinates of the state influence domain, the safety boundary range corresponding to the state space dimension is retrieved from the industrial control semantic knowledge base. When the boundary coordinates exceed the safety boundary range, the state influence domain is determined to be out of bounds. For sub-control tasks determined to be out of bounds, a pre-protection flag is set, a safety protection task template is extracted from the industrial control semantic knowledge base, a predecessor protection node is generated for the out-of-bounds sub-control task, and a mandatory constraint relationship is established. A task constraint relationship network is generated by using the sub-control tasks in the task execution sequence and the predecessor protection node as network nodes, and the dependency relationship and the mandatory constraint relationship as connection relationship.

[0011] A second aspect of the present invention provides an intelligent console control system based on multimodal interaction, comprising: The data acquisition unit is used to acquire multimodal input data from different interaction channels of the industrial console; The semantic fusion unit is used to extract modal semantic features from the input data of each modality, map the semantic features of each modality to a unified semantic space, obtain fused semantic features, and identify control intentions. The conflict disambiguation unit is used to detect semantic conflicts in the control intent based on a pre-built industrial control semantic knowledge base and perform disambiguation processing to obtain the disambiguated intent expression. The task decomposition unit is used to decompose the intent expression into multiple sub-control tasks, define the pre-state conditions and post-state changes of each sub-control task, solve the state transition path from the current state to the target state through backpropagation of the target state constraints, select the optimal state transition path and generate a task execution sequence. The task scheduling unit is used to establish a mapping relationship between the task execution sequence and the device state space, identify the implicit dependencies between each sub-control task by solving the state variable constraints in the mapping relationship, perform topological rearrangement of the task execution sequence and allocate execution time windows and target execution devices, and generate a control instruction sequence. The execution feedback unit is used to send the control command sequence to the corresponding device of the industrial console, drive the device to perform operations, and collect device execution status feedback information.

[0012] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0014] This invention solves the problem of semantic inconsistency between different interaction channels by unifying and fusing multimodal input data through semantic mapping. It combines industrial control semantic knowledge base for conflict detection and disambiguation, eliminating control ambiguity caused by contradictions or fuzziness in multimodal inputs, significantly improving the accuracy and robustness of intent recognition, ensuring that the semantics of control commands are clear and unambiguous, avoiding equipment misoperation, and greatly reducing the risk of production accidents.

[0015] The control intent is automatically decomposed into sub-control tasks with clear preconditions and postconditions. The optimal state transition path is searched by backpropagation target state constraints. The path selection fully considers feasibility, cost and efficiency. The generated sub-task execution sequence achieves the control objective with minimal state change cost. While ensuring the quality of task completion, resource consumption is reduced and the efficiency and adaptability of industrial control process planning are improved.

[0016] The task execution sequence is finely mapped to the device state space. Based on state variable constraints, it automatically identifies implicit dependencies between subtasks, rearranges the sequence topology, and rationally allocates execution time windows and target devices. This effectively avoids resource competition, state conflicts, or execution deadlocks, improves the parallelism and execution smoothness of multi-device collaborative operations, shortens the control task completion time, collects device execution status feedback to form a closed loop, and enhances the real-time adaptive adjustment capability of the control system. It is particularly suitable for dynamic and complex industrial environments. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the intelligent console control method based on multimodal interaction according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the method for extracting modal semantic features and identifying control intentions according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0020] Figure 1 This is a flowchart illustrating the intelligent console control method based on multimodal interaction according to an embodiment of the present invention. The present invention provides an intelligent console control method based on multimodal interaction, comprising: Acquire multimodal input data from different interaction channels of the industrial console; Modal semantic features are extracted from the input data of each modality, and the semantic features of each modality are mapped to a unified semantic space to obtain fused semantic features and identify control intentions; Based on a pre-built industrial control semantic knowledge base, semantic conflicts in the control intent are detected and disambiguated to obtain the disambiguated intent expression. The intent expression is decomposed into multiple sub-control tasks. The pre-state conditions and post-state changes of each sub-control task are defined. The state transition path from the current state to the target state is solved by backpropagation of the target state constraints. The optimal state transition path is selected and the task execution sequence is generated. A mapping relationship between the task execution sequence and the device state space is established. The implicit dependencies between each sub-control task are identified by solving the state variable constraints in the mapping relationship. The task execution sequence is topologically rearranged and the execution time window and target execution device are allocated to generate a control instruction sequence. The control command sequence is sent to the corresponding device on the industrial control console to drive the device to perform operations and collect device execution status feedback information.

[0021] Figure 2 This is a flowchart illustrating the method for extracting modal semantic features and identifying control intentions according to an embodiment of the present invention. Modal semantic features are extracted from the input data of each modality, and these features are mapped to a unified semantic space to obtain fused semantic features and identify control intentions. The process includes: Modality-specific encoding is performed on the input data of each modality to extract semantic features. Semantic decomposition is performed to obtain multiple semantic sub-units. The semantic similarity between semantic sub-units of different modalities is calculated. Based on the semantic similarity, cross-modal semantic sub-unit pairs representing the same semantic content are identified and used as semantic pairs. Spatial alignment constraints are applied to the semantic pairs and mapped to a unified semantic space. Semantic clusters are formed by clustering the semantic sub-units in the unified semantic space based on the spatial distance between them. The weighted average of the semantic sub-units in each semantic cluster is then used to obtain the cluster representation vector. Semantic role labeling is performed on each cluster representation vector to identify the corresponding control object role, control action role, and control parameter role. The cluster representation vectors of different roles are then decoded into control object identifier, control action type, and control parameter value, respectively. A triplet structure is constructed based on the semantic dependency relationship between the control object identifier, the control action type, and the control parameter value. The control object identifier is the subject, the control action type is the predicate, and the control parameter value is the object. The triplet structure is used as the control intent.

[0022] After acquiring multimodal input data from different interaction channels of the industrial control console, modality-specific encoding is required for each modality's data characteristics to extract its respective semantic features. For the speech modality, a deep neural network-based acoustic model is used to convert the original audio frame sequence into phoneme-level feature representations, which are then decoded by a language model to obtain word sequence-level semantic feature vectors. For the text modality, a pre-trained language representation model is used to embed words and encode context in the input text, obtaining context semantic vectors corresponding to each word. For gesture or touch modality, convolutional neural networks or graph neural networks are used to perform temporal modeling of keypoint sequences or touch trajectories to extract action semantic features. After each modality is encoded, semantic decomposition is performed on the semantic feature sequence of each modality, dividing the continuous feature sequence into several discrete semantic subunits. Each semantic subunit corresponds to a local semantic fragment, such as an operational verb phrase, a parameter value range, or a target device reference. The division of semantic subunits is completed based on a semantic boundary detection algorithm, which determines the segmentation position by detecting semantic abrupt changes in the feature vector sequence.

[0023] After obtaining the semantic sub-unit sets for each modality, the semantic similarity between semantic sub-units of different modalities is calculated to identify cross-modal semantic correspondences. Let modality... The Middle The feature vectors of the semantic subunits are Modality The Middle The feature vectors of the semantic subunits are Then the semantic similarity between the two Calculated using cosine similarity: ,

[0024] when Exceeding the preset similarity threshold At that time, Semantic pairs are identified as cross-modal semantic subunits representing the same semantic content. By traversing all pairwise subunit combinations across all modalities, a complete set of semantic pairs can be constructed, providing a foundation for subsequent cross-modal alignment. This process effectively addresses the semantic gap between different signal forms used by different modalities to express the same control intent.

[0025] Spatial alignment constraints are imposed on the identified semantic pairs to map the semantic sub-units of each modality to a unified semantic space. The core idea of ​​spatial alignment constraints is that the Euclidean distance between two sub-units belonging to the same semantic pair in the unified semantic space should be as small as possible, while the distance between sub-units not belonging to the same pair should be kept sufficiently large. By optimizing the alignment loss function, the projection matrix of each modality feature vector to the unified semantic space is learned. and , so that the mapped vector and In a unified semantic space, they are close to each other. After mapping, the unified semantic space contains semantic sub-unit representations from all modalities. These representations share the same coordinate system and can be directly compared in terms of spatial distance and clustered.

[0026] In a unified semantic space, semantic clusters are formed by clustering semantic sub-units based on the spatial distance between them. Clustering algorithms employ density-based or distance-based methods to group semantically similar sub-units into the same cluster. Each semantic cluster represents a semantic concept that is repeatedly expressed or mutually corroborated in multimodal input. A weighted average of the semantic sub-units within each semantic cluster is then performed to obtain the cluster representation vector. Weighting coefficients The value is determined based on the reciprocal of the distance between each sub-unit and the cluster center. The closer the sub-unit is to the cluster center, the greater its contribution to the cluster representation vector. The calculation method is as follows: ,

[0027] in Indicates the first A set of sub-unit indices in a semantic cluster For the first The th cluster Vector representation of each subunit in a unified semantic space These are the corresponding normalized weight coefficients. Cluster representation vector. By integrating information from all sub-units within the semantic cluster, it exhibits stronger semantic robustness than a single modality and can effectively suppress the interference of single-modal noise on semantic understanding.

[0028] For each cluster representation vector Semantic role labeling identifies the type of role a character plays in expressing control intent. Semantic roles are categorized into three types: control object roles, control action roles, and control parameter roles. Control object roles correspond to the specific equipment or equipment component being manipulated in an industrial control scenario, such as a pump, a valve, or a production line. Control action roles correspond to the type of operation applied to the control object, such as starting, stopping, adjusting, or switching. Control parameter roles correspond to the specific values ​​or ranges required for the execution of control actions, such as rotational speed, temperature setpoint, or pressure threshold. Semantic role labeling is achieved through a role classifier, which uses clustered vector representations. Given the input, output the corresponding role category probability distribution, and take the category with the highest probability as the semantic role label of the cluster.

[0029] After role labeling is completed, the cluster representation vectors of different roles are input into their respective decoders to be decoded into specific control semantic symbols. The cluster representation vector of the control object role is mapped to a control object identifier via the object decoder, which corresponds to the device address or device number in the industrial control system. The cluster representation vector of the control action role is mapped to standardized control action type enumeration values ​​via the action decoder. The cluster representation vector of the control parameter role is mapped to specific control parameter values ​​via the parameter decoder, along with physical unit information. The decoding process relies on a specialized vocabulary and mapping rules specific to the industrial control field to ensure that the decoding results conform to the standard format of the industrial control protocol.

[0030] Based on the semantic dependencies between the controlled object identifier, the control action type, and the control parameter value, a triple structure is constructed. The semantic dependencies of the triples are as follows: the control action type must be compatible with the device type corresponding to the controlled object identifier, i.e., the device supports the action; the control parameter value must be within the allowed parameter range of the control action type, and the physical unit must be consistent with the device interface specification. Using the controlled object identifier as the subject, the control action type as the predicate, and the control parameter value as the object, a structured triple representation is constructed, in the form of (device identifier, action type, parameter value). This triple structure, as a standardized expression of control intent, can clearly describe the complete control semantics of "which device, what operation to perform, and with what parameters," providing structured input for subsequent semantic conflict detection and task decomposition. In actual industrial scenarios, a single multimodal interaction will generate multiple triples, corresponding to multiple parallel control intents expressed by the user simultaneously. Each triple is independent or has a certain logical relationship, and all are managed uniformly in the subsequent processing stage.

[0031] Based on a pre-built industrial control semantic knowledge base, semantic conflicts in the control intent are detected and disambiguated to obtain the disambiguated intent expression, including: Using the control object identifier, control action type, and control parameter value in the control intent as semantic nodes, the semantic dependency relationship between each semantic node is retrieved from the industrial control semantic knowledge base, and directed connection edges are established between each semantic node to construct the intent semantic graph based on the semantic dependency relationship. Traverse the directed connection edges in the intent semantic graph to extract the semantic consistency conditions marked on the edges, substitute the semantic nodes connected by the directed connection edges into the semantic consistency conditions for verification, and mark the directed connection edges as conflicting edges and record the source and target nodes connected by the conflicting edges. For conflict edges, the set of replaceable nodes for the source node and the set of replaceable nodes for the target node are retrieved from the industrial control semantic knowledge base. The set of replaceable nodes is searched for node replacement combinations that meet the semantic consistency condition. The node replacement combination with the smallest sum of semantic distances to the original node is selected for replacement. The replaced nodes are used to update the intent semantic graph and the attribute values ​​of each node are extracted. The extracted attribute values ​​are recombined into a triple structure in the order of control object identifier, control action type and control parameter value. The recombined triple structure is used as the disambiguated intent expression.

[0032] After obtaining the fused semantic features and identifying the control intent, semantic conflict detection and disambiguation processing are required to ensure the accuracy and consistency of subsequent control commands. In industrial control scenarios, the control intent expressed by operators through various means such as voice, gestures, and touch often contains implicit physical constraints and equipment state dependencies. Directly converting the recognition results into control commands can lead to problems such as parameter out-of-bounds errors, contradictory action sequences, or equipment state conflicts. Therefore, based on a pre-built industrial control semantic knowledge base, the control intent is structurally modeled, and conflict detection and disambiguation replacement are performed on the graph structure.

[0033] The industrial control semantic knowledge base stores a large number of domain-specific semantic dependency rules, covering the correspondence between equipment types and executable actions, the legal value range of control parameters, physical constraints between actions and parameters, and mutual exclusion or cooperative relationships between different controlled objects. Using the control object identifier, control action type, and control parameter value in the control intent as semantic nodes, the semantic dependencies between these nodes are retrieved from the knowledge base. For example, for a variable frequency drive, its control object identifier corresponds to the equipment node in the knowledge base, "speed increase" as the control action type corresponds to the action node, and the target speed value as the control parameter value corresponds to the parameter node. The knowledge base records the "support" dependency between equipment nodes and action nodes, and the "parameter constraint" dependency between action nodes and parameter nodes. These dependencies carry semantic consistency conditions, such as "the target speed must not exceed the rated speed limit" or "the speed increase action is only valid when the equipment is in operation." Based on the retrieved semantic dependencies, directed connections are established between the semantic nodes, with the direction of the edges indicating the direction of dependency propagation, thereby constructing an intent semantic graph describing the current control intent.

[0034] After the semantic graph is built, all directed edges in the graph are traversed, and the semantic consistency conditions labeled on each edge are extracted. The attribute values ​​of the source and target nodes connected by the edge are then substituted into the semantic consistency conditions for verification. Semantic consistency conditions can be numerical range constraints, state enumeration constraints, or logical combination constraints. Taking numerical range constraints as an example, if the condition labeled on a certain edge requires control over the numerical value of a parameter... satisfy ,in and The lower and upper bounds of the parameter recorded in the knowledge base are used to substitute the current control parameter value into the inequality for judgment. If the verification passes, the edge remains in a normal state; if the verification fails, the directed connection edge is marked as a conflict edge, and the source node identifier and target node identifier connected to the conflict edge are recorded simultaneously for locating the range of nodes that need to be replaced during subsequent disambiguation processing. For logical combination constraints, such as "when the action type is 'emergency stop,' the control parameter value must be zero," verification is performed through conditional logic judgment. After all directed connection edges have been traversed, the set of conflict edges represents the location where there is a semantic contradiction in the current control intent.

[0035] For each conflict edge, the sets of replaceable nodes for the source node and the target node are retrieved from the industrial control semantic knowledge base. The sets of replaceable nodes consist of candidate nodes in the knowledge base that have semantically similar meanings to the original node but different attribute values. For example, if the target node is a speed parameter value node that exceeds the rated range, its set of replaceable nodes includes all legal speed parameter value nodes under that equipment type. The Cartesian product space of the replaceable node sets is searched for node replacement combinations that satisfy the semantic consistency condition on the conflict edge. To maintain the semantic integrity of the intent while satisfying the constraints, the combination with the smallest sum of semantic distances to the original node needs to be selected from all replacement combinations that satisfy the condition. Let the semantic embedding vector of the original source node be... The semantic embedding vector of the original target node is The semantic embedding vector of the source node in the candidate replacement combination is The semantic embedding vector of the target node is Then the sum of semantic distances Defined as: ,

[0036] Select to make The minimum possible replacement combination is chosen as the final replacement solution, thereby eliminating semantic conflicts while preserving the semantic content of the original control intent to the greatest extent possible. If neither the source node nor the target node of a conflicting edge has a replacement node that satisfies the consistency condition, the conflicting edge and its associated nodes are marked as undisambiguated nodes, and disambiguation failure information is fed back to the upper-level logic, triggering a manual confirmation process.

[0037] After replacing all conflicting edges, the intent semantic graph is updated using the replaced nodes. The semantic consistency of all directed connections in the graph is then rechecked to confirm that no more conflicting edges exist. After the update, the attribute values ​​of each node in the intent semantic graph are extracted, including the device number or device type identifier of the node corresponding to the control object identifier, the action category code of the control action type node, and the specific value and unit information of the control parameter value node. These attribute values ​​are then recombined into a triple structure in a fixed order of control object identifier, control action type, and control parameter value, in the form of (device number, action category, parameter value). This triple structure represents the disambiguated intent expression, possessing a clear semantic consistency guarantee, and can be directly used as input for subsequent task decomposition and state transition path solving.

[0038] In real-world industrial scenarios, a single control intent may involve multiple controlled objects and multiple action-parameter pairs. Therefore, the intent semantic graph typically contains multiple subgraph structures, each corresponding to an independent control sub-intent. The aforementioned conflict detection and disambiguation process is executed independently for each subgraph, ultimately merging the disambiguation results of all subgraphs to generate a list of intent expressions containing multiple triples, serving as the complete disambiguated intent expression output. This structured triple representation facilitates the subsequent decomposition of the intent into sub-control tasks, defining pre-state conditions and post-state changes for each sub-task, thereby supporting the advancement of the complete control flow.

[0039] The intent expression is decomposed into multiple sub-control tasks. Preconditions and subsequent state changes are defined for each sub-control task. The state transition path from the current state to the target state is solved through backpropagation of target state constraints. The optimal state transition path is selected, and a task execution sequence is generated, including: Dependency parsing is performed on the intent expression to identify the dominance relationship between the control action type and the control object identifier. Based on the dominance relationship, the control action type is split into the main control action and the auxiliary control action, which are respectively corresponding to sub-control tasks. The current state vector is constructed by reading the device status register of the industrial console and the target state vector is constructed by inferring the expected operating state based on the control parameter values. After projecting the current state vector and the target state vector onto a unified state space, the state coordinate offset caused by each sub-control task in the unified state space is calculated as the subsequent state change. The state coordinates before the offset are used as the pre-state conditions of each sub-control task. Starting from the target state vector, a reverse search is performed along the state gradient direction. Each search selects the sub-control task that makes the state coordinates closer to the current state vector and records it and its corresponding state coordinates as search nodes. When a search node enters the neighborhood of the current state vector, the reverse search terminates. Connect each search node to form a state transition path. Calculate the direction and magnitude changes of adjacent state coordinates in each state transition path to calculate the path smoothness. Select the state transition path with the highest path smoothness as the optimal state transition path. Extract the corresponding sub-control tasks according to the reverse order of the search nodes in the optimal state transition path and arrange them sequentially to generate a task execution sequence.

[0040] After obtaining the disambiguated intent expression, it needs to be further decomposed into executable sub-control tasks, and the optimal execution path from the current device state to the target state needs to be planned. The intent expression is essentially a structured semantic description containing elements such as control actions, controlled objects, and related parameters. When performing dependency parsing on the intent expression, a dependency tree is used as the basis to identify the dominance relationship between predicates and objects, i.e., the dominance relationship between the control action type and the controlled object identifier. For example, in the intent expression "raise the reactor temperature to the target value and keep it stable," "raise to" is the primary dominance action, and "keep it stable" is the secondary dominance action, each corresponding to an independent control semantic level. Based on the hierarchy of the dominance relationship, the control action type is split into primary control actions and secondary control actions, with each type of action corresponding to a sub-control task, thus forming a set of sub-control tasks.

[0041] Read the device status registers of the industrial control console to obtain the current operating parameters of each controlled device, including measured values ​​of physical quantities such as temperature, pressure, flow rate, and rotational speed. Arrange these parameters in a predefined dimensional order to construct the current state vector. Based on the control parameter values ​​carried in the intent expression, combined with the equipment operation rules and process constraints, the expected operating state that each controlled device should reach after completing the control objective is deduced. The target values ​​of each parameter in the expected operating state are arranged in order of the same dimension to construct the target state vector. The current state vector and the target state vector together define the starting and ending points of the state planning problem, providing boundary conditions for solving the subsequent state transition path.

[0042] The current state vector and target state vector Projecting onto a unified state space, the unified state space eliminates dimensional differences by normalizing parameters in each dimension, making different physical quantities comparable under the same coordinate system. For each sub-control task, the state coordinate offset caused by its execution in the unified state space is calculated and denoted as the post-state change vector. ,in This is the index of the sub-control task number. The state coordinates before the offset represent the prerequisite state conditions for this sub-control task, indicating that the sub-task can only be legally triggered when the device is in the state corresponding to those coordinates. The subsequent state change vector... The calculation depends on the action type and control parameters of the sub-control task, and is obtained by querying the equipment action state change model stored in the industrial control semantic knowledge base.

[0043] Backward search from the target state vector Starting from the coordinates in the unified state space, gradually move towards the current state vector along the direction of the state gradient. Approaching. In each search step, traverse all sub-control tasks and select their subsequent state change vector. Enables the current search coordinates Towards Sub-control tasks that are closer in direction, i.e., selection The smallest sub-control task. The selected sub-control task and its corresponding state coordinates. Record it as a search node, update the search coordinates to the state coordinates of that node, and proceed to the next search iteration.

[0044] When the state coordinates of the search node are compared with the current state vector Euclidean distance between Falling into the preset neighborhood radius Inner time, that is Terminate the reverse search process, neighborhood radius The value of is set comprehensively based on the measurement accuracy of the device status parameters and the allowable control error to ensure that the difference between the state coordinates at the end of the search and the actual current state is within an acceptable range. Reverse search can reach the neighborhood range from different intermediate paths, thus forming multiple candidate state transition paths. Connecting the search nodes in each candidate path sequentially yields several paths from Nearby Nearby state transition paths.

[0045] Path smoothness is calculated for each candidate state transition path to evaluate the execution coherence of the path in the state space. The path smoothness is calculated by analyzing the state coordinate changes between adjacent search nodes in the path and extracting the directional change angle between adjacent state coordinate offset vectors. And the ratio of amplitude change ,in This refers to the numbering of adjacent node pairs in the path. Direction change angle. Reflecting the degree of transition of the path in the state space, the magnitude of the change is greater than Reflects the degree of change in control intensity between adjacent steps. Path smoothness. Defined as the reciprocal of the combined penalty value of the ratio of the direction change angle to the amplitude change ratio of all adjacent node pairs, the smaller the direction change angle and the closer the amplitude change ratio is to 1, the higher the path smoothness, indicating that the task execution process corresponding to the path is more stable and will not cause drastic fluctuations in the device state.

[0046] Select path smoothness The highest-ranking candidate state transition path is selected as the optimal state transition path. The search proceeds in reverse order of the nodes in the optimal state transition path, starting from those closest to the desired state transition path. The nodes move sequentially towards each other. The corresponding sub-control tasks are extracted from the node directions and arranged in the extraction order to generate a task execution sequence. The order of each sub-control task in this execution sequence corresponds to its actual execution order in the equipment, ensuring that the equipment starts from the current state and gradually transitions to the target state along the optimal path. The generated task execution sequence will serve as input for the subsequent control command generation stage, used to establish the mapping relationship between sub-control tasks and the equipment state space, thereby identifying implicit dependencies and generating the final control command sequence to drive each target device in the industrial control console to complete the operation in the planned order.

[0047] A mapping relationship is established between the task execution sequence and the device state space. Implicit dependencies between sub-control tasks are identified by solving the state variable constraints in the mapping relationship. The task execution sequence is topologically rearranged, and execution time windows and target execution devices are allocated to generate a control instruction sequence, including: The state space dimension definition corresponding to each sub-control task in the task execution sequence is retrieved from the device state model library. The control parameter values ​​are projected onto the state space dimension to form state vector coordinates. The modification range of each sub-control task is recorded as the state influence domain. Calculate the intersection of the state influence domains of the preceding and subsequent sub-control tasks and determine the dependencies. Read the boundary safety constraints from the industrial control semantic knowledge base to verify whether the state influence domains have exceeded the limits and mark the sub-control tasks that need to be protected in advance. Construct a task constraint relationship network. Identify the strongly connected components in the task constraint relationship network and merge them into atomic execution units, then rearrange the task execution sequence in ascending order of node hierarchy depth; Obtain historical execution time statistics for each sub-control task after rearrangement and calculate conservative duration estimates based on the statistical data. Sequentially determine the execution time window for each sub-control task in a time cursor progressive manner. Based on the execution time window of each sub-control task, query the device topology diagram to obtain executable device nodes, evaluate the task queue length of each device node and select the shortest one as the target execution device, encode the sub-control task into the target device instruction message and generate a control instruction sequence according to the timing.

[0048] In generating control command sequences, it is necessary to establish a precise mapping relationship between the task execution sequence and the equipment state space. The state space dimension definition corresponding to each sub-control task in the task execution sequence is retrieved from the equipment state model library. This definition describes the state space coordinate axes and their physical meanings corresponding to key parameters (such as speed, temperature, pressure, valve opening, etc.) during equipment operation. The control parameter values ​​contained in each sub-control task are projected onto the corresponding state space coordinate axes according to the dimension definition, forming state vector coordinates, thereby transforming discrete control parameter operations into coordinate point representations in the state space. Simultaneously, the state space dimension range involved in the execution of each sub-control task is recorded, i.e., the set of parameters that change due to the task and their variation range. This range is defined as the state influence domain of the sub-control task. Precise characterization of the state influence domain is the foundation for subsequent dependency analysis; it reflects the "footprint" of each sub-control task in the equipment state space.

[0049] Calculate the intersection of the state influence domains of the preceding and succeeding sub-control tasks. If the state influence domains of the two tasks have a non-empty intersection, it indicates that the state variables operated by the succeeding task share or cover the state variables of the preceding task, thus determining that there is a data dependency or resource contention dependency between them. Let the task... The state influence domain is ,Task The state influence domain is ,like Then add from the task constraint relationship network. arrive The system identifies directed dependency edges. After determining dependencies, it reads boundary safety constraints associated with each state space dimension from the industrial control semantic knowledge base to verify whether the state influence domain of each sub-control task will cause the corresponding parameters to exceed the safe operating boundary. If the state influence domain of a sub-control task touches the safety boundary, it is marked as a task requiring pre-protection, and a pre-protection node is added to it in the task constraint relationship network to ensure that necessary safety preprocessing operations (such as limit locking, pressure relief, interlock confirmation, etc.) are completed before the task is executed. Through the above process, a complete task constraint relationship network is constructed, which expresses the explicit and implicit dependencies between all sub-control tasks in the form of a directed graph.

[0050] Strongly connected components in the task constraint network are identified. Strongly connected components are subsets of tasks in the network that are interdependent and cannot be executed independently. The Tarjan algorithm is used to perform a depth-first traversal of the task constraint network to identify all strongly connected components. Multiple sub-control tasks within each strongly connected component are merged into an atomic execution unit. Tasks within an atomic execution unit must be executed as a whole and indivisible to avoid inconsistencies in the device state due to partial execution. After merging the strongly connected components, a topological sort is performed on the updated task constraint network (where each node is either an atomic execution unit or an independent sub-control task). The task execution sequence is rearranged according to the increasing depth of the node hierarchy, ensuring that all dependencies satisfy the temporal constraint that "preceding tasks execute before subsequent tasks," thus obtaining an ordered task execution sequence that satisfies all dependency constraints.

[0051] Obtain historical execution time statistics for each rearranged sub-control task (or atomic execution unit). This historical execution time data comes from the industrial console's execution log database, containing multiple actual execution time records for this type of task under different devices and load conditions. Based on the historical statistical data, calculate a conservative time estimate for each sub-control task. Specifically, use the mean of the historical execution times plus a certain number of standard deviations to provide an upper bound estimate, ensuring sufficient redundancy in the time window allocation and avoiding timing conflicts in subsequent tasks due to underestimating execution times. Let the... The historical average execution time of each sub-control task is The standard deviation is Then its conservative duration estimate ,in The preset safety margin coefficient is typically between 1.5 and 3.0, depending on the reliability requirements of the industrial site. A time cursor progressive method is used to sequentially determine the execution time window for each sub-control task: the initial time cursor is set to the current time. For the first task in the topology sequence, the start time of its execution time window is the initial time cursor, and the end time is the start time plus a conservative estimate of the task's duration. After allocating the time window for the current task, the time cursor is advanced to the end time of that task's time window, and the minimum interval required for task switching is added to the cursor. Then, a time window is allocated for the next task, and so on, until all tasks have completed their time window allocation.

[0052] Based on the allocated execution time windows for each sub-control task, the set of executable device nodes capable of executing the task is queried from the device topology diagram. The device topology diagram describes the connection relationships, functional types, and current online status of each device node in the industrial control console. By matching the functional requirements of the sub-control task with the functional labels of the device nodes, candidate device nodes that meet the functional conditions and are available within the corresponding time window are selected. The current task queue length of the candidate device nodes is evaluated; the task queue length reflects the backlog of tasks currently awaiting execution on the device node. The device node with the shortest task queue length is selected as the target execution device to achieve load balancing and reduce task waiting latency. After selecting the target execution device, the operation parameters, execution time windows, precondition verification requirements, and security constraint information of the sub-control task are encoded according to the communication protocol format supported by the target device to generate the target device command message. The instruction messages of all sub-control tasks are arranged sequentially according to the time order after topology rearrangement to form a complete control instruction sequence. Each instruction message in this sequence carries a clear target device identifier, execution start and end timestamps, and priority markers to ensure that the industrial console can drive each device to complete the corresponding operation in sequence, on time, and accurately during the instruction issuance phase.

[0053] Calculate the intersection of the state influence domains of the preceding and subsequent sub-control tasks and determine their dependencies. Read boundary safety constraints from the industrial control semantic knowledge base to verify whether the state influence domains have exceeded their limits and mark sub-control tasks requiring prior protection. Construct a task constraint relationship network, including: The sub-control tasks in the task execution sequence are paired up, and the boundary coordinates of the state influence domain of the preceding and subsequent sub-control tasks in each task pair are extracted. The overlapping intervals are calculated in each dimension of the state space and combined to form the intersection region of the state influence domain. Determine whether the volume of the intersection region of the state influence domain is zero. If the volume is non-zero, determine that the subsequent sub-control task depends on the preceding sub-control task and record the dependency relationship. Based on the boundary coordinates of the state influence domain, the safety boundary range corresponding to the state space dimension is retrieved from the industrial control semantic knowledge base. When the boundary coordinates exceed the safety boundary range, the state influence domain is determined to be out of bounds. For sub-control tasks determined to be out of bounds, a pre-protection flag is set, a safety protection task template is extracted from the industrial control semantic knowledge base, a predecessor protection node is generated for the out-of-bounds sub-control task, and a mandatory constraint relationship is established. A task constraint relationship network is generated by using the sub-control tasks in the task execution sequence and the predecessor protection node as network nodes, and the dependency relationship and the mandatory constraint relationship as connection relationship.

[0054] After completing the initial arrangement of the task execution sequence, further analysis of the state influence relationships between each sub-control task is needed to ensure that no state conflicts or safety out-of-bounds errors occur during task execution. All sub-control tasks in the task execution sequence are paired up, and for each task pair, the state influence domain boundary coordinates of the preceding and succeeding sub-control tasks are extracted. The state influence domain boundary coordinates describe the upper and lower bounds of the state space region affected by a sub-control task during execution, specifically represented as the boundary of the hyperrectangular region formed by the minimum and maximum values ​​in each dimension of the state space. For the preceding sub-control task, its state influence domain is determined by the range of changes in all state variables that may change during the execution of the task; for the succeeding sub-control task, its state influence domain boundary coordinates are extracted in the same way.

[0055] After obtaining the boundary coordinates of the state influence domains of the two tasks, the overlapping portion of the two intervals is calculated in each dimension of the state space. For the _th interval in the state space... Let there be a dimension, and let the influence range of the preceding sub-control task in this dimension be . The influence range of subsequent sub-control tasks in this dimension is: Then the overlap interval between the two in this dimension is .like If there is no overlap in that dimension, the overlap length is recorded as zero. The overlapping intervals in all dimensions are combined to form the intersection region of the state influence domains, which is a multidimensional hyperrectangle.

[0056] Determine if the volume of the intersection region of the obtained state influence domains is zero. The volume of the intersection region is calculated by multiplying the overlap lengths of each dimension. If the overlap length of any dimension is zero, the overall volume is zero, indicating that the state influence domains of the two tasks do not share a common region in the state space, and there is no state-level dependency between them. If the volume is non-zero, it means that the state influence domains of the two tasks overlap, and the execution result of the subsequent sub-control task is affected by the execution state of the preceding sub-control task. In this case, it is determined that the subsequent sub-control task depends on the preceding sub-control task, and this dependency is recorded in the dependency table. The dependency entry includes the preceding task identifier, the subsequent task identifier, and the corresponding intersection region description information for use in subsequent topology sorting and time window allocation.

[0057] After identifying dependencies, the safety of the state influence domain boundary coordinates of each sub-control task is further verified. The safety boundary ranges corresponding to each dimension of the current state space are retrieved from the industrial control semantic knowledge base. The industrial control semantic knowledge base stores the physical safety constraints of various industrial equipment and controlled objects under different operating scenarios, including the allowable operating ranges of state variables such as temperature, pressure, speed, current, and displacement. For the first... There are several dimensions, and the security boundary range stored in the knowledge base is defined as follows: If the state influence domain of a certain sub-control task has an upper bound on this dimension. Exceed or the lower realm Below Then, determine the state influence domain of the sub-control task in the th order. A dimensional out-of-bounds event has occurred. If any dimension exceeds its bounds, the overall state influence domain of the sub-control task is determined to be out of bounds, requiring pre-emptive protection.

[0058] For sub-control tasks determined to be out of bounds, a pre-protection flag is set in their task record. The flag includes the out-of-bounds dimension number and the out-of-bounds direction (above the upper boundary or below the lower boundary). Then, a safety protection task template matching the out-of-bounds situation is extracted from the industrial control semantic knowledge base. The safety protection task template is a predefined standardized operation sequence used to adjust relevant state variables to a safe range or activate corresponding protection mechanisms (such as current limiting protection, overvoltage protection, over-temperature alarm interlock, etc.) before executing control tasks that may trigger safety out-of-bounds actions. Based on the specific out-of-bounds dimension and amount of the out-of-bounds sub-control task, the most matching safety protection task template is selected from the template library to generate an independent precursor protection node for that out-of-bounds sub-control task. The state influence domain of the precursor protection node is set to the range of state changes required to correct the out-of-bounds dimension; its execution result ensures that the state variables are within the safe boundary when subsequent out-of-bounds sub-control tasks are executed. In the task relationship record, a mandatory constraint relationship is established between the out-of-bounds sub-control task and its corresponding predecessor protection node. The mandatory constraint relationship requires that the predecessor protection node must be executed before the out-of-bounds sub-control task starts, and it is not allowed to be skipped or parallelized by any scheduling strategy.

[0059] After dependency identification and safety boundary crossing handling are completed, a task constraint relationship network is constructed using all sub-control tasks and all generated predecessor protection nodes in the task execution sequence as network nodes, and dependencies and mandatory constraints as directed connection edges. The connection edges corresponding to dependencies point from the preceding sub-control task to the following sub-control task, expressing the temporal constraint that "the following task can only start after the preceding task is completed." The connection edges corresponding to mandatory constraints point from the predecessor protection node to the boundary-crossing sub-control task, expressing that "the completion of the protection node's execution is a hard prerequisite for the execution of the boundary-crossing task." Each connection edge in the task constraint relationship network carries a constraint type label to distinguish between ordinary state dependency constraints and safety mandatory constraints, facilitating higher priority processing of mandatory constraints during the subsequent topology reordering stage.

[0060] After the task constraint network is constructed, it is topologically sorted to ensure that all dependencies and mandatory constraints are satisfied. The topology sorting uses an in-degree-based iterative approach, selecting nodes with an in-degree of zero and adding them to the execution queue each time, while updating the in-degree of their successor nodes, until all nodes are added to the execution queue. If a circular dependency occurs during the sorting process, a conflict alarm is triggered, and the process is reverted to the intent disambiguation stage for reprocessing. The sorting result, combined with conservative time estimates for each sub-control task (determined by the historical average execution time and safety margin), allocates an execution time window to each node, ultimately forming a complete sequence of control instructions that satisfies both safety and dependency constraints, which is then sent to the corresponding devices on the industrial control console for execution.

[0061] A second aspect of the present invention provides an intelligent console control system based on multimodal interaction, comprising: The data acquisition unit is used to acquire multimodal input data from different interaction channels of the industrial console; The semantic fusion unit is used to extract modal semantic features from the input data of each modality, map the semantic features of each modality to a unified semantic space, obtain fused semantic features, and identify control intentions. The conflict disambiguation unit is used to detect semantic conflicts in the control intent based on a pre-built industrial control semantic knowledge base and perform disambiguation processing to obtain the disambiguated intent expression. The task decomposition unit is used to decompose the intent expression into multiple sub-control tasks, define the pre-state conditions and post-state changes of each sub-control task, solve the state transition path from the current state to the target state through backpropagation of the target state constraints, select the optimal state transition path and generate a task execution sequence. The task scheduling unit is used to establish a mapping relationship between the task execution sequence and the device state space, identify the implicit dependencies between each sub-control task by solving the state variable constraints in the mapping relationship, perform topological rearrangement of the task execution sequence and allocate execution time windows and target execution devices, and generate a control instruction sequence. The execution feedback unit is used to send the control command sequence to the corresponding device of the industrial console, drive the device to perform operations, and collect device execution status feedback information.

[0062] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0063] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0064] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method for an intelligent console based on multimodal interaction, characterized in that, include: Acquire multimodal input data from different interaction channels of the industrial console; Modal semantic features are extracted from the input data of each modality, and the semantic features of each modality are mapped to a unified semantic space to obtain fused semantic features and identify control intentions; Based on a pre-built industrial control semantic knowledge base, semantic conflicts in the control intent are detected and disambiguated to obtain the disambiguated intent expression. The intent expression is decomposed into multiple sub-control tasks. The pre-state conditions and post-state changes of each sub-control task are defined. The state transition path from the current state to the target state is solved by backpropagation of the target state constraints. The optimal state transition path is selected and the task execution sequence is generated. A mapping relationship between the task execution sequence and the device state space is established. The implicit dependencies between each sub-control task are identified by solving the state variable constraints in the mapping relationship. The task execution sequence is topologically rearranged and the execution time window and target execution device are allocated to generate a control instruction sequence. The control command sequence is sent to the corresponding device on the industrial control console to drive the device to perform operations and collect device execution status feedback information.

2. The method according to claim 1, characterized in that, Modal semantic features are extracted from the input data of each modality, and the semantic features of each modality are mapped to a unified semantic space to obtain fused semantic features and identify control intentions, including: Modality-specific encoding is performed on the input data of each modality to extract semantic features. Semantic decomposition is performed to obtain multiple semantic sub-units. The semantic similarity between semantic sub-units of different modalities is calculated. Based on the semantic similarity, cross-modal semantic sub-unit pairs representing the same semantic content are identified and used as semantic pairs. Spatial alignment constraints are applied to the semantic pairs and mapped to a unified semantic space. Semantic clusters are formed by clustering the semantic sub-units in the unified semantic space based on the spatial distance between them. The weighted average of the semantic sub-units in each semantic cluster is then used to obtain the cluster representation vector. Semantic role labeling is performed on each cluster representation vector to identify the corresponding control object role, control action role, and control parameter role. The cluster representation vectors of different roles are then decoded into control object identifier, control action type, and control parameter value, respectively. A triplet structure is constructed based on the semantic dependency relationship between the control object identifier, the control action type, and the control parameter value. The control object identifier is the subject, the control action type is the predicate, and the control parameter value is the object. The triplet structure is used as the control intent.

3. The method according to claim 1, characterized in that, Based on a pre-built industrial control semantic knowledge base, semantic conflicts in the control intent are detected and disambiguated to obtain the disambiguated intent expression, including: Using the control object identifier, control action type, and control parameter value in the control intent as semantic nodes, the semantic dependency relationship between each semantic node is retrieved from the industrial control semantic knowledge base, and directed connection edges are established between each semantic node to construct the intent semantic graph based on the semantic dependency relationship. Traverse the directed connection edges in the intent semantic graph to extract the semantic consistency conditions marked on the edges, substitute the semantic nodes connected by the directed connection edges into the semantic consistency conditions for verification, and mark the directed connection edges as conflicting edges and record the source and target nodes connected by the conflicting edges. For conflict edges, the set of replaceable nodes for the source node and the set of replaceable nodes for the target node are retrieved from the industrial control semantic knowledge base. The set of replaceable nodes is searched for node replacement combinations that meet the semantic consistency condition. The node replacement combination with the smallest sum of semantic distances to the original node is selected for replacement. The replaced nodes are used to update the intent semantic graph and the attribute values ​​of each node are extracted. The extracted attribute values ​​are recombined into a triple structure in the order of control object identifier, control action type and control parameter value. The recombined triple structure is used as the disambiguated intent expression.

4. The method according to claim 1, characterized in that, The intent expression is decomposed into multiple sub-control tasks. Preconditions and subsequent state changes are defined for each sub-control task. The state transition path from the current state to the target state is solved through backpropagation of target state constraints. The optimal state transition path is selected, and a task execution sequence is generated, including: Dependency parsing is performed on the intent expression to identify the dominance relationship between the control action type and the control object identifier. Based on the dominance relationship, the control action type is split into the main control action and the auxiliary control action, which are respectively corresponding to sub-control tasks. The current state vector is constructed by reading the device status register of the industrial console and the target state vector is constructed by inferring the expected operating state based on the control parameter values. After projecting the current state vector and the target state vector onto a unified state space, the state coordinate offset caused by each sub-control task in the unified state space is calculated as the subsequent state change. The state coordinates before the offset are used as the pre-state conditions of each sub-control task. Starting from the target state vector, a reverse search is performed along the state gradient direction. Each search selects the sub-control task that makes the state coordinates closer to the current state vector and records it and its corresponding state coordinates as search nodes. When a search node enters the neighborhood of the current state vector, the reverse search terminates. Connect each search node to form a state transition path. Calculate the direction and magnitude changes of adjacent state coordinates in each state transition path to calculate the path smoothness. Select the state transition path with the highest path smoothness as the optimal state transition path. Extract the corresponding sub-control tasks according to the reverse order of the search nodes in the optimal state transition path and arrange them sequentially to generate a task execution sequence.

5. The method according to claim 1, characterized in that, A mapping relationship is established between the task execution sequence and the device state space. Implicit dependencies between sub-control tasks are identified by solving the state variable constraints in the mapping relationship. The task execution sequence is topologically rearranged, and execution time windows and target execution devices are allocated to generate a control instruction sequence, including: The state space dimension definition corresponding to each sub-control task in the task execution sequence is retrieved from the device state model library. The control parameter values ​​are projected onto the state space dimension to form state vector coordinates. The modification range of each sub-control task is recorded as the state influence domain. Calculate the intersection of the state influence domains of the preceding and subsequent sub-control tasks and determine the dependencies. Read the boundary safety constraints from the industrial control semantic knowledge base to verify whether the state influence domains have exceeded the limits and mark the sub-control tasks that need to be protected in advance. Construct a task constraint relationship network. Identify the strongly connected components in the task constraint relationship network and merge them into atomic execution units, then rearrange the task execution sequence in ascending order of node hierarchy depth; Obtain historical execution time statistics for each sub-control task after rearrangement and calculate conservative duration estimates based on the statistical data. Sequentially determine the execution time window for each sub-control task in a time cursor progressive manner. Based on the execution time window of each sub-control task, query the device topology diagram to obtain executable device nodes, evaluate the task queue length of each device node and select the shortest one as the target execution device, encode the sub-control task into the target device instruction message and generate a control instruction sequence according to the timing.

6. The method according to claim 5, characterized in that, Calculate the intersection of the state influence domains of the preceding and subsequent sub-control tasks and determine their dependencies. Read boundary safety constraints from the industrial control semantic knowledge base to verify whether the state influence domains have exceeded their limits and mark sub-control tasks requiring prior protection. Construct a task constraint relationship network, including: The sub-control tasks in the task execution sequence are paired up, and the boundary coordinates of the state influence domain of the preceding and subsequent sub-control tasks in each task pair are extracted. The overlapping intervals are calculated in each dimension of the state space and combined to form the intersection region of the state influence domain. Determine whether the volume of the intersection region of the state influence domain is zero. If the volume is non-zero, determine that the subsequent sub-control task depends on the preceding sub-control task and record the dependency relationship. Based on the boundary coordinates of the state influence domain, the safety boundary range corresponding to the state space dimension is retrieved from the industrial control semantic knowledge base. When the boundary coordinates exceed the safety boundary range, the state influence domain is determined to be out of bounds. For sub-control tasks determined to be out of bounds, a pre-protection flag is set, a safety protection task template is extracted from the industrial control semantic knowledge base, a predecessor protection node is generated for the out-of-bounds sub-control task, and a mandatory constraint relationship is established. A task constraint relationship network is generated by using the sub-control tasks in the task execution sequence and the predecessor protection node as network nodes, and the dependency relationship and the mandatory constraint relationship as connection relationship.

7. A multimodal interaction-based intelligent console control system, used to implement the method as described in any one of claims 1-6, characterized in that, include: The data acquisition unit is used to acquire multimodal input data from different interaction channels of the industrial console; The semantic fusion unit is used to extract modal semantic features from the input data of each modality, map the semantic features of each modality to a unified semantic space, obtain fused semantic features, and identify control intentions. The conflict disambiguation unit is used to detect semantic conflicts in the control intent based on a pre-built industrial control semantic knowledge base and perform disambiguation processing to obtain the disambiguated intent expression. The task decomposition unit is used to decompose the intent expression into multiple sub-control tasks, define the pre-state conditions and post-state changes of each sub-control task, solve the state transition path from the current state to the target state through backpropagation of the target state constraints, select the optimal state transition path and generate a task execution sequence. The task scheduling unit is used to establish a mapping relationship between the task execution sequence and the device state space, identify the implicit dependencies between each sub-control task by solving the state variable constraints in the mapping relationship, perform topological rearrangement of the task execution sequence and allocate execution time windows and target execution devices, and generate a control instruction sequence. The execution feedback unit is used to send the control command sequence to the corresponding device of the industrial console, drive the device to perform operations, and collect device execution status feedback information.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images