Humanoid robot-oriented dialogue arrangement method and related equipment

Abstract

The embodiment of the invention provides a humanoid robot-oriented dialogue arrangement method and related equipment, and belongs to the technical field of robots and artificial intelligence. The method comprises the steps that operation instruction input facing a target robot is obtained, a semantic track is constructed, the semantic track indicates a sequence of actions needing to be executed and execution conditions needing to be met when each action in the sequence of actions needing to be executed is executed, multi-source constraint data of a manufacturing site where the target robot is located are obtained, and a manufacturing state track is constructed; the manufacturing state rail indicates the operation constraint and the safety constraint of the manufacturing site; according to the manufacturing state orbit, performing step-by-step registration on each action in the sequence of actions needing to be executed indicated by the semantic orbit, and generating a candidate executable action sequence; and performing logic rehearsal based on the candidate executable action sequence, and determining an actual execution action sequence of the target robot according to a logic rehearsal result. According to the embodiment of the invention, the security risk caused when semantic reasonability and manufacturing state reasonability cannot be met at the same time during instruction execution can be effectively reduced.

Description

A dialogue orchestration method and related equipment for humanoid robots Technical Field

[0001] This application relates to the fields of robotics and artificial intelligence, and in particular to a dialogue orchestration method and related equipment for humanoid robots. Background Technology

[0002] In recent years, large language models have demonstrated enormous potential in the field of robot task planning and execution, becoming a key technology connecting natural language instructions with physical world operations. In industrial production scenarios, the importance of this technology is increasingly prominent: it can understand high-level, ambiguous natural language instructions (such as "assemble the next batch of product A") issued by operators or scheduling systems and decompose them into a series of understandable and serializable specific task steps. Through initial integration with robot perception and control systems, large language models have initially achieved a mapping from "semantic intent" to "action sequence." However, when connecting the large language model at the dialogue layer with the constraints of the manufacturing layer (MES / PLC / SOP), it generally relies on empirical rules or offline reachability assessments, lacking the ability to dynamically verify natural language instructions online. This results in semantically reasonable instructions being executed even in abnormal manufacturing states such as violating interlocks or crossing recipe / workstation switching windows, thus posing safety risks.

[0003] In summary, the problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a dialogue orchestration method and related equipment for humanoid robots. Through the registration of semantic and manufacturing states in a unified manner and multiple verifications of logical pre-calculation, online dynamic verification of natural language instructions and manufacturing states is achieved, effectively reducing the security risks caused by the inability to simultaneously satisfy semantic rationality and manufacturing state rationality during instruction execution.

[0005] To achieve the above objectives, one aspect of this application proposes a dialogue orchestration method for humanoid robots. The method includes: acquiring operation command input for a target robot; constructing a semantic track based on the operation command input; the semantic track indicating a sequence of actions to be executed and the execution conditions that each action in the sequence of actions to be executed must satisfy; acquiring multi-source constraint data of the manufacturing site where the target robot is located; constructing a manufacturing state track based on the multi-source constraint data; the manufacturing state track indicating the operation constraints and safety constraints of the manufacturing site; performing stepwise registration on each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, determining the actions in the sequence of actions to be executed that meet preset requirements, so as to generate a candidate executable action sequence; performing logical pre-calculation based on the candidate executable action sequence, and determining the actual execution action sequence of the target robot based on the result of the logical pre-calculation; the logical pre-calculation is performed on the interlock triggering conditions, timing correctness, and resource consumption of each action in the executable action sequence.

[0006] In some embodiments, constructing a semantic track based on the operation instruction input includes: performing semantic parsing based on the operation instruction to obtain an original action sequence; the original action sequence includes several actions and annotation information corresponding to each action; the annotation information includes the object, parameters, execution conditions, and uncertainty measure corresponding to each action; arranging the original action sequence in chronological order to obtain a sequence of actions to be executed; and constructing the semantic track based on the sequence of actions to be executed and the annotation information corresponding to each action in the sequence of actions to be executed.

[0007] In some embodiments, acquiring multi-source constraint data of the manufacturing site where the target robot is located, and constructing a manufacturing state track based on the multi-source constraint data, includes: acquiring work orders, process flow, and recipe information from the manufacturing execution system; acquiring input signals, output signals, and equipment interlock and emergency stop states from the programmable logic controller; acquiring operational constraint information from standard operating procedure documents; obtaining the multi-source constraint data based on the data acquisition results; compiling the multi-source constraint data based on a preset layering method; and constructing the manufacturing state track based on the compilation results; the preset layering includes an equipment layer, a workstation layer, a recipe layer, and an interlock layer.

[0008] In some embodiments, the semantic track performs stepwise registration of each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track to determine the actions in the sequence of actions to be executed that meet preset requirements, including: determining the annotation information corresponding to the target action in the sequence of actions to be executed indicated by the semantic track; the annotation information includes the object, parameters, execution conditions, and uncertainty measure corresponding to the target action; calculating the structural compatibility of the target action based on the annotation information corresponding to the manufacturing state track and the target action; calculating the constraint satisfaction of the target action based on the safety constraints indicated by the manufacturing state track and the execution conditions corresponding to the target action; calculating the safety margin of the target action based on the operational constraints indicated by the manufacturing state track and the execution conditions corresponding to the target action; calculating the registration score corresponding to the target action based on the structural compatibility, constraint satisfaction, safety margin, and uncertainty measure corresponding to the target action; if the registration score corresponding to the target action is greater than a preset threshold, the target action is determined to be an action that meets the preset requirements.

[0009] In some embodiments, if the registration score corresponding to the target action is not greater than the preset threshold, the method further includes: calculating a counterfactual patch set for the action based on the manufacturing state track and the identification information corresponding to the target action; and adjusting the identification information corresponding to the target action according to the counterfactual patch set.

[0010] In some embodiments, the step of performing logical pre-calculation based on the candidate executable action sequence and determining the actual execution action sequence of the target robot based on the result of the logical pre-calculation includes: constructing a shadow memory space based on the candidate executable action sequence; the shadow memory space is used to simulate the process of executing actions in the candidate executable action sequence in an offline state; performing logical pre-calculation on each action in the candidate executable action sequence based on the shadow memory space, determining whether the execution of the action satisfies its corresponding interlock triggering condition, timing correctness, and resource consumption, and determining the actions in the candidate executable action sequence that meet the requirements based on the determination, so as to determine the actual execution action sequence of the target robot.

[0011] In some embodiments, the method further includes: based on the actual action sequence, controlling the target robot to execute each action in the actual action sequence node by node and obtaining the device data corresponding to the action at each node; if the device data corresponding to the current node is abnormal, then reverting to the previous node to re-execute the corresponding action and returning the reason for the abnormality.

[0012] To achieve the above objectives, another aspect of this application proposes a dialogue orchestration system for humanoid robots. The system includes: a data acquisition module, configured to acquire operation command inputs for a target robot and construct a semantic track based on the operation command inputs; the semantic track indicates a sequence of actions to be executed and the execution conditions that each action in the sequence must satisfy; acquire multi-source constraint data of the manufacturing site where the target robot is located and construct a manufacturing state track based on the multi-source constraint data; the manufacturing state track indicates the operation constraints and safety constraints of the manufacturing site; an analysis module, configured to perform stepwise registration of each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, determine the actions in the sequence of actions to be executed that meet preset requirements, and generate a candidate executable action sequence; and a simulation module, configured to perform logical pre-calculation based on the candidate executable action sequence, and determine the actual execution action sequence of the target robot based on the results of the logical pre-calculation; the logical pre-calculation is performed on the interlock triggering conditions, timing correctness, and resource consumption of each action in the executable action sequence.

[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a dialogue orchestration method, apparatus, electronic device, and storage medium for humanoid robots. This solution constructs a semantic track by acquiring operator input commands to the humanoid robot and constructs a manufacturing state track by acquiring multi-source constraint data of the humanoid robot in the manufacturing site. The semantic track indicates the sequence of actions to be executed corresponding to the operation commands and the execution conditions of each action in the sequence, corresponding to "what to do." The manufacturing state track indicates the operational and safety constraints of the manufacturing site, corresponding to "what can be done." Then, based on the manufacturing state track, each action in the sequence of actions to be executed indicated by the semantic track is progressively registered to determine the sequence of actions that meet the requirements. The method of this invention constructs a candidate executable action sequence by performing actions that meet the requirements. The method sets up a first-level gating system, namely "semantic-manufacturing state dual-track consistency," which performs quantitative verification of "semantic intent → permitted state transition" for each action to be executed through registration, thereby determining which instructions are semantically reasonable and state-permitted. Furthermore, the method also sets up a second-level gating system with logical pre-calculation, which further verifies the candidate executable action sequence through pre-calculation. This dual gating system intercepts instructions that are "semantically reasonable but not permitted by the manufacturing state," reducing the probability of false triggering, unauthorized access, and chain reactions from the source, effectively reducing security risks caused by the inability to simultaneously satisfy semantic reasonableness and manufacturing state reasonableness during instruction execution. Attached Figure Description

[0016] Figure 1 is a flowchart of a dialogue orchestration method for humanoid robots provided in an embodiment of this application; Figure 2 is a flowchart of another dialogue orchestration method for humanoid robots provided in an embodiment of this application; Figure 3 is a structural diagram of a dialogue orchestration system for humanoid robots provided in an embodiment of this application; Figure 4 is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0020] 1) MES (Manufacturing Execution System): This is the core information system connecting the enterprise's planning layer and the shop floor control layer. It monitors and manages the elements of people, machines, materials, methods, and environment on the production site in real time, and is specifically responsible for issuing production orders, scheduling jobs, controlling processes, tracing quality, tracking materials, collecting data, and analyzing performance.

[0021] 2) PLC (Programmable Logic Controller): It is the "core control brain" of industrial automation systems. It is a rugged computer designed for harsh industrial environments. It receives input signals from sensors and buttons, performs logical judgments and mathematical calculations according to the user's pre-written program (ladder diagram, etc.), and then drives actuators (such as motors, valves, indicator lights) to complete the specified actions.

[0022] 3) SOP (Standard Operating Procedure): This is a detailed and standardized set of work instructions for repetitive operations. It clearly defines the steps, methods, quality and safety standards, and responsibilities for performing specific tasks or processes, aiming to ensure that different personnel can achieve consistent, reliable, safe, and efficient results when operating.

[0023] 4) OPC UA (Open Platform Unified Communication Architecture): This is a new generation, cross-platform industrial data communication and information modeling standard. It completely solves the limitations of traditional OPC, is independent of hardware platforms and operating systems, and integrates powerful built-in security mechanisms. Its core lies not only in transmitting data, but also in providing machines, equipment, and systems with "understandable" information containing contextual semantics through service-oriented architecture (SOA) and information modeling capabilities, achieving enterprise-level secure and reliable interoperability from the shop floor to the cloud.

[0024] 5) Linear Temporal Logic (LTL) is a mathematical logic used to describe and verify the evolution of the behavior of concurrent systems (such as software and hardware protocols) over time. It introduces timing operators such as "the future is always / eventually / until" into classical logic to rigorously describe the time-dependent properties of event sequences (such as "a request must eventually receive a response").

[0025] 6) Metric-based sequential logic (MTL): This is an enhanced extension of time-based sequential logic (LTL), which introduces a precise "metric" concept into time constraints. By binding specific time intervals (such as "within the next 5 seconds" or "until 1 to 3 seconds") to timing operators, MTL can provide a rigorous quantitative real-time description and verification of system behavior (e.g., "an emergency signal must be responded to within 100 milliseconds").

[0026] In recent years, large language models have been used for robot task understanding and step generation, and are gradually being integrated with the perception-control link to complete the mapping from natural language to executable actions. Meanwhile, industrial production has long relied on Manufacturing Execution Systems (MES), Programmable Logic Controllers (PLCs / Safety PLCs), OPCUA information models, and the IEC 61131-3 language family to express workstation states, process recipes, interlocks, and safety constraints. These two technological systems often operate independently in engineering: the dialogue layer excels at semantic understanding and intent decomposition, while the manufacturing layer emphasizes discrete event-driven processes and state machine constraints. This separation leads to a systematic misalignment between "linguistically reasonable" plans and "site-permitted" state transitions, especially in areas such as interlocks, energy isolation, workstation restricted areas, and recipe switching windows. The semantic layer lacks executability judgment and evidentiary feedback channels, resulting in both false trigger risks and audit blind spots.

[0027] To address the aforementioned gaps, academia and industry have explored various approaches: First, using language-based high-level step planning or language-to-skill selection / code generation to score offline skill libraries based on accessibility or utility; second, using end-to-end vision-language-action models to transfer internet or teaching experience into strategies; and third, adhering to the hierarchical modeling and communication specifications of ISA-95, OPCUA, and IEC61131-3 to achieve system integration. However, these solutions generally fail to progressively align and verify dialogue intent with the online manufacturing state machine, and even more so, they lack a "verifiable before execution" endorsement mechanism on the controller side. When semantic plans violate interlocks or parameters are missing, they are often rejected based on empirical hints or static rules, making it difficult to trace back to specific workstation states, PLC variables, or SOP clauses, and lacking operational guidance on "how to correct at the lowest cost before allowing execution."

[0028] In real production lines, timing and real-time issues further amplify these deficiencies. PLC scan cycles, fieldbus jitter, and network latency often cause theoretically feasible language plans to fail in terms of implementation timing; while MES / PLC mislabeling and sensor drift can cause state perception biases. The lack of reliable decisions and auditable records on the controller side makes it difficult for humanoid robots to establish a reliable closed loop in the interaction-planning-execution chain, affecting both cycle efficiency and meeting quality and safety audit requirements.

[0029] In view of this, this application provides a dialogue orchestration method, apparatus, electronic device, and storage medium for humanoid robots. This solution constructs a semantic track by acquiring operator input commands to the humanoid robot and a manufacturing state track by acquiring multi-source constraint data of the humanoid robot at the manufacturing site. The semantic track indicates the sequence of actions to be executed corresponding to the operation commands and the execution conditions of each action in the sequence, corresponding to "what to do." The manufacturing state track indicates the operational and safety constraints at the manufacturing site, corresponding to "what can be done." Then, based on the manufacturing state track, each action in the sequence of actions to be executed indicated by the semantic track is progressively registered to determine the actions in the sequence that meet the requirements. The method constructs candidate executable action sequences. The invention employs a first-level gating system, namely "semantic-manufacturing state dual-track consistency," which quantifies the semantic intent to allow state transitions for each action through registration, thereby determining which instructions are semantically reasonable and state-permitted. Furthermore, the invention includes a second-level gating system with logical pre-calculation, which further verifies the candidate executable action sequences through pre-calculation. This dual gating system intercepts instructions that are "semantically reasonable but not permitted by the manufacturing state," reducing the probability of false triggering, unauthorized access, and chain reactions from the source, effectively mitigating security risks caused by the inability to simultaneously satisfy semantic and manufacturing state requirements during instruction execution.

[0030] The dialogue orchestration method for humanoid robots provided in this application relates to the fields of humanoid robot and artificial intelligence technology. This dialogue orchestration method for humanoid robots can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the dialogue orchestration method for humanoid robots, but is not limited to the above forms.

[0031] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0032] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0033] Figure 1 is an optional flowchart of a dialogue orchestration method for humanoid robots provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S100 to S300.

[0034] Step S100: Obtain the operation command input for the target robot, and construct a semantic track based on the operation command input; the semantic track indicates the sequence of actions to be executed and the execution conditions that must be met when each action in the sequence of actions is executed; obtain the multi-source constraint data of the manufacturing site where the target robot is located, and construct a manufacturing state track based on the multi-source constraint data; the manufacturing state track indicates the operation constraints and safety constraints of the manufacturing site.

[0035] The method in this application is based on the idea of ​​"semantic-manufacturing dual-track consistency". It constructs the semantic track and the manufacturing track for dialogue with the humanoid robot in parallel within a unified operation cycle. The semantic track is constructed according to the operator's instructions, and the manufacturing track is constructed according to the multi-source constraint data of the manufacturing site, so as to prepare the data foundation for subsequent analysis.

[0036] Step S200: Based on the manufacturing state track, each action in the sequence of actions to be executed indicated by the semantic track is progressively registered to determine the actions in the sequence of actions to be executed that meet the preset requirements, so as to generate a candidate sequence of executable actions.

[0037] Based on the manufacturing state track, the sequence of actions to be executed indicated by the semantic track is registered step by step. The actions to be executed at each step are quantitatively verified according to "semantic intent → allowed state transition". Safety fences can also be introduced as necessary constraints to participate in gating. Finally, candidate executable action sequences are determined through gating analysis.

[0038] Step S300: Perform logical pre-calculation based on candidate executable action sequences, and determine the actual execution action sequence of the target robot based on the results of the logical pre-calculation; the logical pre-calculation is performed on the interlock triggering conditions, timing correctness and resource usage of each action in the executable action sequence.

[0039] The method of the present invention also sets up a second level of gating, which performs logical pre-calculation based on the candidate executable action sequence, calculates and verifies the candidate executable actions, simulates their execution, and sees whether they meet the real logical calculation of their interlocking, timing and resource consumption.

[0040] In step S100 of some embodiments, before constructing the semantic track and manufacturing state track, the method of this embodiment first receives production instructions issued by the operator in the form of voice or text, and automatically captures the on-site context within the same time window, including work order number, workstation number, recipe version, batch identifier, and key status bits from the PLC / safety PLC. To ensure that subsequent judgments are reproducible, all inputs are assigned a session ID and a unified timestamp, and are recorded together with the hash fingerprint of the source system to form a minimal and complete context message. On the voice path, edge noise reduction and speaker separation are performed, and on the text path, terminology regularization is performed to stably map on-site jargon to standard action words, avoiding semantic drift from affecting the execution layer.

[0041] After context binding, the system measures and caches timing-related channel characteristics, including PLC scan cycle, fieldbus jitter, and network round-trip latency. These real-time metrics then participate in the adaptive adjustment of gating thresholds to ensure that a "theoretically feasible" plan does not fail in real-world timing. Once this stage is complete, the message is submitted to the parallel pipeline for semantic track generation and manufacturing machine extraction, initiating core computation for dual-track consistency.

[0042] In some embodiments, step S100 involves constructing a semantic track based on operation instruction input, including but not limited to steps S111 to S113.

[0043] Step S111: Perform semantic parsing based on the operation instructions to obtain the original action sequence; the original action sequence includes several actions and the annotation information corresponding to each action; the annotation information includes the object, parameters, execution conditions and uncertainty measure corresponding to each action; arrange the original action sequence in chronological order to obtain the action sequence to be executed.

[0044] Under the constraints of the conversational context, natural language instructions are parsed into a sequence of action primitives arranged in chronological order, and each action is labeled with its object, parameters, and preconditions / postconditions. The system simultaneously calculates the confidence and distribution entropy of intent, object, and key parameters, and explicitly assigns uncertainty labels to ambiguous or missing slots for adaptive convergence of the subsequent consistency threshold. For new terms not yet logged in, the system employs a restricted expansion strategy: a temporary mapping based on similarity and context is performed, but the uncertainty weight of the action is significantly increased to avoid direct approval when evidence is insufficient.

[0045] Step S112: Construct a semantic track based on the sequence of actions to be executed and the annotation information corresponding to each action in the action queue to be executed.

[0046] A semantic track is constructed based on the sequence of actions to be performed and the annotation information corresponding to each action in the sequence.

[0047] Specifically, to facilitate alignment with the manufacturing state track, the semantic track retains the reversible mapping from natural language to action primitives and the pre-assertion inference chain during generation. For example, when the "ignition" intention is parsed, the system automatically infers and includes pre-assertions such as "torque verification passed," "emergency stop not triggered," and "safety light curtain cleared." These assertions are used for subsequent consistency calculations and will also generate operator-oriented evidentiary explanations during blocking or clarification, supporting a traceable and auditable interactive experience.

[0048] In some embodiments, step S100 involves acquiring multi-source constraint data of the manufacturing site where the target robot is located, and constructing a manufacturing trajectory based on the multi-source constraint data. This process includes, but is not limited to, steps S121 to S122: Step S121 involves acquiring work orders, process flow, and recipe information from the manufacturing execution system, acquiring input signals, output signals, and equipment interlock and emergency stop status from the programmable logic controller, acquiring operational constraint information from the standard operating procedure document, and obtaining multi-source constraint data based on the data acquisition results.

[0049] Work orders, process flow, and recipe constraints are retrieved from the MES system, and discrete I / O, interlock, and emergency stop statuses are retrieved from the PLC / safety PLC. The SOP document is then parsed at the clause level to obtain multi-source constraint data as the data foundation for subsequent analysis.

[0050] Step S122: Compile the multi-source constraint data based on the preset layering method, and construct the manufacturing state track according to the compilation results; the preset layering includes equipment layer, workstation layer, recipe layer and interlock layer.

[0051] Multi-source information is uniformly compiled into a hierarchical model of "equipment-workstation-recipe-interlock," and a labeled finite state machine is formed at the workstation granularity. The reachable edges record trigger conditions, interlock strength, safety preconditions, and expected readback signals. In the face of vendor heterogeneity and proprietary extensions, the system extracts ladder diagrams / functional block diagrams into standardized Boolean and timing constraints through adapters to ensure consistent expression across devices.

[0052] When changes in the state space are detected due to work order switching, recipe version upgrades, or equipment maintenance, the system only performs incremental refreshes on the affected subgraphs and publishes the new version and effective window. The unchanged parts retain the old pointers to avoid execution interruption caused by rebuilding the entire graph. The semantic track and all subsequent decision outputs reference the state graph version number and hash at that time, so that the auditing process can reproduce the decision scenario "according to the knowledge at that time" to ensure the closure of the evidence chain.

[0053] In some embodiments, step S200 involves the semantic track progressively registering each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, and determining the actions in the sequence of actions to be executed that meet the preset requirements. This process includes, but is not limited to, steps S210 to S240: Step S210: Based on the semantic track, determine the annotation information corresponding to the target action in the action process to be executed indicated by it; the annotation information includes the object, parameters, execution conditions, and uncertainty measure corresponding to the target action.

[0054] The annotation information corresponding to each target action in the action process to be executed is determined based on the semantic track, which serves as the data basis for subsequent registration.

[0055] Step S220: Calculate the structural compatibility of the target action based on the annotation information corresponding to the manufacturing state track and the target action; calculate the constraint satisfaction of the target action based on the safety constraints indicated by the manufacturing state track and the execution conditions corresponding to the target action; calculate the safety margin of the target action based on the operational constraints indicated by the manufacturing state track and the execution conditions corresponding to the target action.

[0056] Register the semantic track sequentially on the manufacturing state track (reachable state diagram). Calculate three types of indicators for each action: structural compatibility, constraint satisfaction, and safety margin, which serve as the data basis for subsequent analysis.

[0057] Step S230: Based on the structural compatibility, constraint satisfaction, and safety margin of the target action, as well as the uncertainty metric corresponding to the target action, a registration score is calculated to obtain the registration score corresponding to the target action. If the registration score corresponding to the target action is greater than a preset threshold, the target action is determined to be an action that meets the preset requirements.

[0058] The system integrates semantic uncertainty to obtain a weighted score S, and uses this score to determine whether the corresponding target action meets preset requirements. The gating strategy can be adjusted to require S and a safety fence as necessary conditions. Specifically, when S is above the upper threshold and there is no hard interlock, the action enters the "candidate executable" category; when S falls into the gray zone, the system switches to the clarification branch; when S is below the lower threshold or triggers a hard interlock, the system immediately blocks the action and prepares a degradation suggestion.

[0059] In some embodiments, to meet the real-time requirements of industrial applications, an adaptive latency / jitter mechanism is introduced to dynamically adjust upper / lower thresholds and waiting windows based on the PLC scan cycle, fieldbus jitter, and network round-trip latency, avoiding the release of processes that are "theoretically feasible but timing-incompatible." Each decision outputs a reason code and referenced evidence (state nodes / transition edges, interlocked variables, and SOP entries), providing a calculable basis for subsequent evidence-based interactions and audit archiving.

[0060] In some embodiments, in scenarios with gray areas and slight inconsistencies, the system solves for a minimum counterfactual patch set on the current state machine trajectory. This patch set modifies the necessary preconditions, parameters, or action sequences with minimal cost and shortest latency, making the target transition legitimate again. The patch set is paraphrased as the shortest correction path easily understood by the operator, and the least critical questions are evaluated using Value of Information (VoI) to reduce round-trip costs. Simultaneously, the gating threshold is adaptively adjusted based on the aforementioned measured PLC scan cycle, bus jitter, and network latency to prevent "paper compliance but implementation failure" due to timing violations.

[0061] In some embodiments, in step S240, if the registration score corresponding to the target action is not greater than a preset threshold, the method of this embodiment further includes: calculating based on the manufacturing state track and the identification information corresponding to the target action to obtain a counterfactual repair set of the action; and adjusting the identification information corresponding to the target action according to the counterfactual repair set.

[0062] When consistency is insufficient (registration score does not meet preset threshold) but it is not hard interlocked, the system will solve for the minimum counterfactual patch set on the reachable state graph constructed by the current state machine trajectory. That is, the set of preconditions, parameters or action sequences that need to be changed with the minimum cost or shortest delay, so that the target transfer edge becomes legal again. The patch set is translated into a correction path that is easy for the operator to understand, such as "first complete the torque review and approval of SOP-3.2, then switch ignition", and includes an assessment of the estimated time consumption, resource consumption and impact on production cycle, to help the site quickly make trade-offs. In some embodiments, in step S300, the process of performing logical pre-calculation based on candidate executable action sequences and determining the actual execution action sequence of the target robot based on the results of logical pre-calculation includes, but is not limited to, steps S310 to S320: Step S310, constructing a shadow memory space based on candidate executable action sequences; the shadow memory space is used to simulate the process of executing the actions in the candidate executable action sequences in an offline state.

[0063] After an action enters the "candidate executable" state, the corresponding control or action sequence is written into the PLC's shadow space (shadow coil / ghost variable). The controller then performs actual interlocking and timing calculations within one or more scan cycles, but does not drive the field devices.

[0064] Step S320: Perform logical pre-calculation on each action in the candidate executable action sequence based on the shadow memory space, determine whether the corresponding interlock triggering condition, timing correctness and resource usage are met when the action is executed, and determine the actions that meet the requirements in the candidate executable action sequence based on the judgment, so as to determine the actual execution action sequence of the target robot.

[0065] By performing logical pre-calculation on each action, it is determined whether the corresponding interlock is satisfied during the execution process. The candidate executable action sequence is further verified through pre-calculation. By using dual gating, instructions that are "semantically reasonable but not allowed by the manufacturing state" are intercepted, reducing the probability of false triggering, unauthorized access and chain destruction from the source, and finally determining the actual execution action sequence of the robot.

[0066] Furthermore, if the calculation passes, the controller generates an executability proof token containing a timestamp, a nonce, a controller identifier, and a state diagram version pointer, and signs the token on the controller side before returning it to the upper layer. If the calculation fails, it returns the reason for failure and the unmet interlock bit, blocking subsequent processes on the real channel. Before issuing real control, the upper layer must verify the token's validity, signature, and session binding relationship. Only after successful verification can the token be written to the real device channel, thus forming a trusted link of "semantic release—controller endorsement—device execution." All tokens and associated input / output snapshots are included in a traceable ledger, supporting fault recovery and clear responsibility boundaries. In unstable network or isolated environments, the module can degenerate into a local short-term caching and delayed verification strategy to ensure a balance between real-time performance and compliance.

[0067] In some embodiments, the method of this embodiment further includes the following steps: Step S410, based on the actual execution action sequence, control the target robot to execute each action in the actual execution action sequence node by node and obtain the device data corresponding to each node when executing the action.

[0068] The actual sequence of actions determined after two-level gating is the production action to be performed by the target robot. The method of this application is based on the rhythm strategy of "micro-step execution + back-read confirmation": each action in the actual sequence of actions is executed node by node, and each node is a micro-step. After each micro-step is executed, the device data after the execution of the node action is required to determine whether the working status of the node is consistent with the expectation.

[0069] Step S420: If the device data corresponding to the current node is abnormal, roll back to the previous node to re-execute the corresponding action and return the reason for the abnormality.

[0070] If the node's readback is abnormal, roll back to the most recent consistent checkpoint and output evidence of the anomaly to prevent error accumulation. For high-risk actions, you can first run the simulation / low-speed trial run channel, and then switch to the standard rate after passing the test to reduce the window for misoperation.

[0071] In some embodiments, a safety fence can be set up to continuously monitor the safety PLC status, energy isolation, collaborative space limits, and humanoid posture boundaries online. Once triggered, the system immediately shuts down and generates state snapshots and evidence fragments before and after the incident. Full-link logs (semantic tracks, state diagram versions, S-curves, shadow verification results, token metadata, execution readbacks) are archived by session / work order / batch. The entire decision-making chain can be replayed in an offline playback tool, allowing for policy recalculation based on different thresholds or weight configurations. This forms a continuous improvement closed loop of "data—policy—execution" and supports quality and safety auditing.

[0072] In summary, the humanoid robot dialogue orchestration method of "semantic-manufacturing dual-track consistency" proposed in this invention aims to achieve a complete closed loop from task input to executable decision output, enabling the robot to determine whether the task is executable while understanding the semantics, and to provide repair and release paths when the conditions are not met.

[0073] First, during the task input phase, the system receives voice or text commands from the operator or the upper-level scheduling system, while automatically acquiring the current manufacturing site context information, including work order number, workstation number, recipe version, PLC and safety PLC status, interlock positions, and safety fence information. Through this process, the system creates a synchronized snapshot of the task's semantic input and manufacturing state, providing a complete data foundation for subsequent semantic parsing and state matching. Second, during the task processing phase, the system establishes a "semantic track" and a "manufacturing state track." The semantic track is responsible for parsing natural language commands, extracting action intent, objects, parameters, and pre- and post-processing conditions; the manufacturing state track constructs finite state machines for equipment and workstations from the MES, PLC, and SOP, clarifying the currently achievable states and safety constraints. The system registers the two tracks through a consistency evaluation mechanism, calculating the task's structural compatibility, constraint satisfaction, and safety margin in the current state. If an instruction is found to be semantically valid but the manufacturing status does not meet the requirements (e.g., violating interlocks or crossing workstation switching windows), the system will automatically solve for the minimum counterfactual repair set, generate a repair plan outlining "how to modify it to make it executable," and provide the operator with natural language prompts to complete or confirm the modification. Finally, during the task output phase, the system will send candidate executable instructions to the PLC shadow space for calculation and verification. If the shadow calculation passes, the controller issues an executability proof token with a timestamp and signature, serving as a prerequisite for action release. Only when semantic consistency and token verification both pass will the instruction be officially issued to the real equipment channel for execution. If verification fails, the system will output the specific reasons and repair suggestions.

[0074] Therefore, the core process of this invention can be summarized as: input task → semantic parsing → manufacturing state verification → repair if unexecutable → release if executable. This mechanism ensures that humanoid robots in intelligent manufacturing scenarios can both understand language intent and comply with actual process and safety rules, achieving a unity of "language feasibility" and "state feasibility".

[0075] The method of this embodiment includes, but is not limited to, the following beneficial effects: (1) Allowing “verifiable” data to be released, which significantly reduces the risk of misoperation.

[0076] By introducing a shadow space on the PLC side and issuing executability proof tokens with timestamps and signatures, the system decentralizes the "executability" decision to the actual logic calculation at the controller scan cycle level; only instructions that "metaphorically consistent + token valid" enter the real channel. This dual gating intercepts instructions that "seem reasonable but are not permitted by the state" from the device, reducing the probability of false triggering, unauthorized execution, and interlocking failure from the source.

[0077] (2) A phased chain of evidence and an auditable closed loop ensure compliance and transparency.

[0078] Each step of granting / blocking will output an evidence card containing "reason code - referenced state diagram version - PLC / safety PLC variable - SOP clause (LTL / MTL counterexample trajectory optional) - Token metadata", which will be archived together with the session ID, timestamp, and version pointer. The judgment process can be reproduced afterward based on "knowledge at the time", meeting the evidence collection needs of quality audits and safety compliance, with clear responsibility boundaries, replayability, and accountability.

[0079] (3) The gray area does not make decisions on a whim; the minimal counterfactual patch set provides "how to do it".

[0080] When consistency is insufficient but not hard interlocked, the system solves the minimum counterfactual patch set on the reachable state graph, and provides a correction path that can be released with the minimum cost / shortest delay (such as supplementary approval, setting key interlocks, adjusting the action sequence). It also selects the fewest and most critical clarification issues using information value (VoI), which significantly reduces back-and-forth communication and downtime waiting time, and improves the first-round clarification hit rate.

[0081] (4) Timing adaptive threshold to ensure that “feasible on paper = feasible on site”.

[0082] The consistency threshold is adaptively adjusted based on the PLC scan cycle, fieldbus jitter, and network RTT; shadow calculation verifies interlocks and timing constraints within the controller cycle, avoiding "theoretical release, execution failure" caused by real-time fluctuations. This mechanism ensures that release decisions consider both logical correctness and timing availability, stabilizing production line cycle time.

[0083] (5) Safety fence + micro-step back reading, quick loss stop and precise reset.

[0084] The execution phase employs a "micro-step execution + readback confirmation" approach, with each micro-step requiring equipment readback or sensor confirmation as the condition for advancement. Once a safety fence is triggered (safety PLC, energy isolation, collaborative space, attitude boundary), the system immediately shuts down and saves snapshots of the window before and after the incident, along with the last valid token. This results in faster fault location, shorter reset steps, and significantly reduced impact radius of misoperations.

[0085] (6) Heterogeneous devices are uniformly expressed and incrementally updated, resulting in low integration and maintenance costs.

[0086] The IEC61131-3 / OPCUA / vendor-specific logic is extracted into a unified workstation-level finite state machine, supporting incremental updates and version management of sub-diagrams during work order switching, recipe changes, and equipment maintenance. The unified state / interlock representation across devices and workstations reduces the workload of integration and modification windows, facilitating large-scale deployment and long-term operation and maintenance.

[0087] (7) Human-machine collaboration is smoother for humanoid robots.

[0088] The semantic track preserves the reversible mapping of natural language-action primitives and the pre-assertion chain, enabling it to "clearly explain why the system interprets it this way, why it is not allowed at this moment, and how it can be allowed" when blocking. This evidence-based dialogue feedback helps frontline staff avoid detours, and combined with the online maintainability of the vocabulary and assertions, it can also quickly adapt to on-site jargon and new processes.

[0089] (8) Continuous learning and strategy recalculation make it smarter the more it is used.

[0090] The entire link log (semantic track, state graph version, S-curve, shadow verification, token, readback) is archived by work order / batch. Offline tools can replay and evaluate the impact of different threshold / weight configurations under the "current version". The adopted repair path will be solidified into a vocabulary mapping and assertion weight, which will improve the first pass rate of similar scenarios in the future and realize the closed-loop optimization of data-policy-execution.

[0091] (9) Both production efficiency and safety are achieved, and downtime and rework are significantly reduced.

[0092] Dual-track consistency gating and shadow tokens prevent errors from reaching the device, while minimal patch sets and VoI clarification compress necessary remediation into the "shortest path," and microstep readback and safety fencing minimize the impact of faults. The overall result is reduced downtime, lower rework rate, and higher first-time success rate, with simultaneous improvements in security and cycle time efficiency.

[0093] (10) Edge-cloud collaboration and isolation elasticity adapt to various deployment forms.

[0094] Semantic parsing and consistency assessment can be completed at the edge, and complex inference or model backup can be migrated to the cloud as needed; token verification and ledger support local short-term caching and delayed reporting when the network is restricted. The system can flexibly trade off between real-time performance, reliability, and compliance, making it suitable for different network and security domain requirements from pilot units to mass production lines.

[0095] Please refer to Figure 2. This application embodiment also provides another dialogue orchestration method for humanoid robots. The method includes, but is not limited to, the following steps: Step 1: Input acquisition and context binding. Receive the operator's voice or text commands and simultaneously capture the context information of the current production site, such as work order number, workstation number, recipe version, PLC status, interlock position and safety fence information, to form a complete task semantic input and manufacturing status snapshot; this information serves as the input basis for subsequent semantic understanding and status alignment.

[0096] Step 2: Semantic Track Generation and Uncertainty Management System parses natural language instructions into action primitive sequences, extracts action objects, parameters, and preconditions, and calculates semantic confidence and uncertainty labels. The parsing results output a structured semantic track, providing a foundation for the next step of registration with the manufacturing state machine.

[0097] Step 3: The manufacturing state machine construction and state diagram update system pulls work orders, processes, interlocks, and SOP constraints from the MES and PLC, compiles them into a workstation-level finite state machine, and incrementally updates it based on work order switching or equipment maintenance. This state diagram reflects the reachable states and safety boundaries on-site and is used for subsequent consistency verification.

[0098] Step 4: The dual-track consistency assessment and gating system gradually registers the semantic track with the manufacturing machine, calculates structural compatibility, constraint satisfaction, and safety margin, and synthesizes them into a consistency score S. If S meets the standard, it enters the candidate execution; if S is insufficient but not hard-interlocked, it solves the minimum counterfactual patch set to generate the minimum cost patching path and clarification problem; if S is too low or triggers interlocking, it directly blocks the process.

[0099] Step 5: PLC Shadow Verification and Token Issuance. When an action enters the candidate executable stage, the system writes the control instructions into the PLC shadow space for real logic and timing calculations. If the calculation passes, the PLC issues an executable proof token with a timestamp, signature, and random number; if the calculation fails, it returns the specific interlock bit and reason for recalculation and repair in the previous stage.

[0100] Step 6: After the micro-step execution and safety fence monitoring verification are passed, the system gradually releases actions according to the "micro-step execution + readback confirmation" strategy, and monitors the safety PLC, energy isolation, and collaborative space boundaries in real time. If the fence is triggered, the system immediately stops and saves a snapshot. The execution results, along with the token, state diagram version, and consistency curve, are archived.

[0101] Step 7: Audit Archiving and Policy Recalculation. The system will uniformly store all process logs (semantic track, state graph version, shadow verification, token metadata, execution readback) in the database, enabling offline replay and policy optimization. Verified and effective patch paths and term mappings will be solidified as new parameters, continuously improving the first-time success rate of subsequent tasks.

[0102] In summary, the method of this embodiment realizes a complete logical closed loop from "understanding whether it can be executed" to "verifying that it can be executed" and then to "secure execution and traceability".

[0103] Please refer to Figure 3. This application embodiment also provides a dialogue orchestration system for humanoid robots, which can implement the above-described method. The system includes: a data acquisition module, used to acquire operation command inputs for the target robot and construct a semantic track based on the operation command inputs; the semantic track indicates the sequence of actions to be executed and the execution conditions that each action in the sequence of actions to be executed must meet; acquire multi-source constraint data of the manufacturing site where the target robot is located, and construct a manufacturing state track based on the multi-source constraint data; the manufacturing state track indicates the operation constraints and safety constraints of the manufacturing site; an analysis module, used to perform stepwise registration of each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, determine the actions in the sequence of actions to be executed that meet the preset requirements, and generate candidate executable action sequences; a simulation module, used to perform logical pre-calculation based on the candidate executable action sequences, and determine the actual execution action sequence of the target robot based on the results of the logical pre-calculation; the logical pre-calculation is performed on the interlock triggering conditions, timing correctness, and resource consumption of each action in the executable action sequence.

[0104] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0105] Referring to Figure 4, this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0106] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0108] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0109] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0110] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] This application provides a dialogue orchestration method, apparatus, electronic device, and storage medium for humanoid robots. The scheme constructs a semantic track by acquiring operator input commands to the humanoid robot and a manufacturing state track by acquiring multi-source constraint data of the humanoid robot in the manufacturing environment. The semantic track indicates the sequence of actions to be executed corresponding to the operation commands and the execution conditions of each action in the sequence, corresponding to "what to do." The manufacturing state track indicates the operational and safety constraints of the manufacturing environment, corresponding to "what can be done." Then, based on the manufacturing state track, each action in the sequence of actions to be executed indicated by the semantic track is progressively registered to determine the actions in the sequence that meet the requirements, thus constructing a... The method constructs a candidate executable action sequence. It establishes a first-level gating system, namely "semantic-manufacturing state dual-track consistency," which quantifies and verifies the semantic intent → allowed state transition for each action through registration, thereby determining which instructions are semantically reasonable and state-permitted. Furthermore, the method also establishes a second-level gating system with logical pre-calculation, which further verifies the candidate executable action sequence through pre-calculation. This dual gating system intercepts instructions that are "semantically reasonable but not permitted by the manufacturing state," reducing the probability of false triggering, unauthorized access, and chain reactions from the source, effectively minimizing security risks caused by the inability to simultaneously satisfy semantic reasonableness and manufacturing state reasonableness during instruction execution.

[0113] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0114] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0117] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0118] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A dialogue orchestration method for humanoid robots, characterized in that, The method includes the following steps: acquiring operation command input for the target robot, and constructing a semantic track based on the operation command input; the semantic track indicates the sequence of actions to be executed and the execution conditions that each action in the sequence of actions to be executed must meet; acquiring multi-source constraint data of the manufacturing site where the target robot is located, and constructing a manufacturing state track based on the multi-source constraint data; the manufacturing state track indicates the operation constraints and safety constraints of the manufacturing site; performing stepwise registration on each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, determining the actions in the sequence of actions to be executed that meet preset requirements, so as to generate a candidate executable action sequence; performing logical pre-calculation based on the candidate executable action sequence, and determining the actual execution action sequence of the target robot based on the result of the logical pre-calculation; the logical pre-calculation is performed on the interlock triggering conditions, timing correctness, and resource consumption of each action in the executable action sequence.

2. The method according to claim 1, characterized in that, The step of constructing a semantic track based on the operation instruction input includes: performing semantic parsing based on the operation instruction to obtain an original action sequence; the original action sequence includes several actions and annotation information corresponding to each action; the annotation information includes the object, parameters, execution conditions, and uncertainty measure corresponding to each action; arranging the original action sequence in chronological order to obtain a sequence of actions to be executed; and constructing the semantic track based on the sequence of actions to be executed and the annotation information corresponding to each action in the sequence of actions to be executed.

3. The method according to claim 1, characterized in that, The process of acquiring multi-source constraint data of the manufacturing site where the target robot is located, and constructing a manufacturing state track based on the multi-source constraint data, includes: acquiring work orders, process flow, and recipe information from the manufacturing execution system; acquiring input signals, output signals, and equipment interlock and emergency stop states from the programmable logic controller; acquiring operational constraint information from the standard operating procedure document; obtaining the multi-source constraint data based on the data acquisition results; compiling the multi-source constraint data based on a preset layering method; and constructing the manufacturing state track based on the compilation results; the preset layering includes an equipment layer, a workstation layer, a recipe layer, and an interlock layer.

4. The method according to claim 1, characterized in that, The semantic track performs stepwise registration of each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, and determines the actions in the sequence of actions to be executed that meet preset requirements. This includes: determining the annotation information corresponding to the target action in the sequence of actions to be executed indicated by the semantic track; the annotation information includes the object, parameters, execution conditions, and uncertainty measure corresponding to the target action; calculating the structural compatibility of the target action based on the annotation information corresponding to the manufacturing state track and the target action; calculating the constraint satisfaction of the target action based on the safety constraints indicated by the manufacturing state track and the execution conditions corresponding to the target action; calculating the safety margin of the target action based on the operational constraints indicated by the manufacturing state track and the execution conditions corresponding to the target action; calculating the registration score corresponding to the target action based on the structural compatibility, constraint satisfaction, safety margin, and uncertainty measure corresponding to the target action; and determining that the target action meets the preset requirements if the registration score corresponding to the target action is greater than a preset threshold.

5. The method according to claim 4, characterized in that, If the registration score corresponding to the target action is not greater than the preset threshold, the method further includes: calculating a counterfactual repair set for the action based on the manufacturing state track and the identification information corresponding to the target action; and adjusting the identification information corresponding to the target action according to the counterfactual repair set.

6. The method according to claim 1, characterized in that, The step of performing logical pre-calculation based on the candidate executable action sequence and determining the actual execution action sequence of the target robot based on the results of the logical pre-calculation includes: constructing a shadow memory space based on the candidate executable action sequence; the shadow memory space is used to simulate the process of executing the actions in the candidate executable action sequence in an offline state; performing logical pre-calculation on each action in the candidate executable action sequence based on the shadow memory space, determining whether the execution of the action satisfies its corresponding interlock triggering condition, timing correctness, and resource consumption, and determining the actions in the candidate executable action sequence that meet the requirements based on the judgment, so as to determine the actual execution action sequence of the target robot.

7. The method according to claim 1, characterized in that, The method further includes: based on the actual action sequence, controlling the target robot to execute each action in the actual action sequence node by node and obtaining the device data corresponding to the action at each node; if the device data corresponding to the current node is abnormal, then reverting to the previous node to re-execute the corresponding action and returning the reason for the abnormality.

8. A dialogue orchestration system for humanoid robots, characterized in that, The system includes: a data acquisition module, used to acquire operation command inputs for the target robot, and construct a semantic track based on the operation command inputs; the semantic track indicates the sequence of actions to be executed and the execution conditions that each action in the sequence of actions to be executed must meet; acquire multi-source constraint data of the manufacturing site where the target robot is located, and construct a manufacturing state track based on the multi-source constraint data; the manufacturing state track indicates the operation constraints and safety constraints of the manufacturing site; an analysis module, used to perform stepwise registration of each action in the sequence of actions to be executed indicated by the semantic track according to the manufacturing state track, determine the actions in the sequence of actions to be executed that meet preset requirements, and generate candidate executable action sequences; a simulation module, used to perform logical pre-calculation based on the candidate executable action sequences, and determine the actual execution action sequence of the target robot based on the results of the logical pre-calculation; the logical pre-calculation is performed on the interlock triggering conditions, timing correctness, and resource consumption of each action in the executable action sequence.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images