An agent and skill architecture-based physical ai brain control system and method

Abstract

The application provides a physical AI brain control system and method based on an Agent and Skill architecture, and belongs to the technical field of industrial automation control and intelligent control of intelligent agriculture. The system comprises AI reasoning cores, real-time motion cores and communication cores which are isolated from each other; the AI reasoning cores carry NPU / GPU to run large models, the real-time motion cores adopt Intel or ARM architecture mainboards to carry NPU / GPU to form main control power AI brain hardware, run Windows, Linux or Hongmeng operating systems, and realize 1ms level motion control, and the communication cores provide CAN, MQTTS / API interfaces and can be mounted with ARM sub-control boards. The software layer comprises a device digital portrait module, a Skill module, a large model task planning Agent, a constraint verification and execution graph compilation module, a real-time execution and safety module and an audit and diagnosis module; the Agent generates a structured control intention, which is compiled into a real-time Skill execution graph with version identification, trigger edges, resource locks, deadline and rollback edges after verification, and is executed deterministically by the real-time core. The visual Skill supports 2D and 3D visual acquisition and processing.

Description

Technical Field

[0001] This invention relates to the fields of industrial automation control, industrial artificial intelligence, real-time bus motion control, and smart agriculture intelligent control technology. More specifically, it relates to a physical AI brain control system and method based on an Agent and Skill architecture. Background Technology

[0002] Currently, programmable logic controllers (PLCs) are the core control equipment in the field of industrial automation. Traditional PLCs rely on ladder diagrams and ST language to write fixed control logic, and implement equipment control through function blocks. They are the core equipment for motion control and I / O interlocking in industrial production lines. However, existing PLC technology has many inherent defects: First, the control logic is rigid and has poor adaptability. Traditional PLC control programs use fixed code. When production line conditions or product specifications change, professional industrial control engineers need to rewrite, debug, and download the program, resulting in extremely low iteration efficiency and an inability to adapt to the production needs of flexible and customized production lines.

[0003] Second, the level of intelligence is low, lacking the ability to build autonomous agents. Traditional PLCs rely entirely on manual programming to solidify logic, unable to autonomously analyze operating conditions, generate control logic, and create intelligent agents through large AI models. While some existing solutions attempt to introduce AI into PLC control—for example, patent publication number WO2021076093A1 proposes equipping PLC function blocks with AI companions—this solution still relies on the PLC program and function blocks as its operational foundation, with AI serving only as an auxiliary capability. It fails to convert upper-level agent decisions into execution diagrams with resource, time, and safety constraints, and to implement deterministic closed-loop execution by an independent hard real-time execution domain. Patent publication number US20250147737A1 generates industrial control code based on natural language, but it is positioned as a programming aid; the generated results still need to be deployed as control programs, failing to address the problem of dynamic task rescheduling during equipment operation.

[0004] Third, the visual inspection capabilities are weak, lacking deep learning iteration capabilities. Traditional PLC-equipped vision devices mostly use traditional threshold and edge algorithms, which can only identify regular defects and cannot identify irregular, subtle, or complex texture defects, and lack the ability for autonomous model learning and optimization; at the same time, there is a millisecond-level delay in communication between the PLC and the camera, and the clock cannot be synchronized, resulting in low accuracy of visual positioning compensation and stuttering in linkage.

[0005] Fourth, EtherCAT / CAN field control and visual analysis, cloud-edge communication, and MES system calls are often deployed separately. External commands may bypass device capability boundaries, security interlocks, and real-time budget constraints. Existing systems lack a general runtime architecture that isolates AI decision-making from hard real-time execution. When visual confidence decreases, the agent becomes unavailable, or communication malfunctions, they cannot switch to a safe state within a defined timeframe.

[0006] In addition, although the control of pumps, valves, fans, and supplemental lighting in smart agriculture can be automated, it often lacks agent / skill orchestration and local security degradation mechanisms that are consistent with industrial control, resulting in insufficient system versatility and scenario adaptability. Summary of the Invention

[0007] The present invention aims to solve the technical problems mentioned in the background art above, and to provide a physical AI brain control system and method based on Agent and Skill architecture.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a physical AI brain control system based on Agent and Skill architecture, which includes a hardware layer and a software layer; The hardware layer includes isolated AI inference cores, real-time motion cores, and communication cores. The AI ​​inference core is equipped with an NPU or GPU and runs large models for task planning and visual inference. The real-time motion core uses an Intel industrial motherboard or an ARM architecture industrial motherboard, equipped with an NPU or GPU to form the main control computing power AI brain hardware, running Windows, Linux, or HarmonyOS operating systems, for multi-axis interpolation, electronic cams, and safety state machines. The communication core provides CAN interface, Gigabit Ethernet interface, USB3 Vision interface, and MQTTS / API interface. The software layer includes: The equipment digital profiling module is used to record equipment topology, slave station identification, axis parameters, I / O points, camera calibration, available skills, and safety state machine; The Skill module is a standardized functional plugin. Each Skill declares the input / output type, resource consumption, maximum execution time, triggering condition, interlocking condition, rollback action, and version number. The large model task planning agent receives natural language instructions or MES / API tasks, selects candidate skills based on the device's digital profile, and generates a structured control intent that includes a set of candidate skills, a deadline, and a fallback level. The constraint verification and execution graph compilation module performs device capability matching, security interlock verification, resource conflict arbitration, and worst-case execution time verification on the control intent in sequence. After the verification is passed, a real-time Skill execution graph with version identifier, trigger edge, resource lock, deadline and rollback edge is generated. The real-time execution and safety module executes the execution diagram at a fixed cycle, drives the slave devices through the EtherCAT master station, and drives the peripheral control board through the CAN interface. The audit and diagnosis module is used to record the control intent, the execution graph version, the skill version, camera timestamp, axis position, slave status and alarms, and publish the permission-filtered audit data through the MQTTS / API interface.

[0009] Furthermore, the real-time motion core has a built-in dual redundant EtherCAT RJ45 interface and uses a DC distributed clock to achieve microsecond-level synchronization of multiple slave stations; the hardware platform of the AI ​​inference core is an ARM industrial motherboard or an Intel X86 industrial motherboard, and the matching GPU computing power module provides 6~50 TOPS of adjustable computing power; the real-time motion core runs Windows, Linux or HarmonyOS operating system, and the fixed period is 0.25ms~10ms.

[0010] Furthermore, the communication core is connected to multiple ARM microcontroller sub-control boards via EtherCAT bus or CAN bus; the main control AI brain is responsible for large model analysis, agent construction, global process scheduling and AI inference, and the ARM sub-control boards are used to perform high-speed IO acquisition, point movement, pulse output, sensor preprocessing, valve group start and stop and actuator closed-loop control locally.

[0011] Furthermore, the Skill module includes a Motion Skill, a Vision Skill, a Sensing I / O Skill, and an Agricultural Skill; the Motion Skill includes point-to-point control, multi-axis interpolation, electronic cam, synchronous following, torque limiting, homing, and soft limit; the Vision Skill includes 2D vision acquisition, 3D vision acquisition, target positioning, size measurement, defect detection, calibration, and coordinate compensation, supporting the access of 2D cameras, 3D cameras (including structured light cameras, binocular cameras, and laser profilometers), and ordinary industrial cameras; the Sensing I / O Skill includes signal reading and writing, digital filtering, threshold alarm, and high-speed counting; the Agricultural Skill includes environmental acquisition, irrigation control, ventilation control, supplemental lighting control, valve interlocking, and pump protection; each Skill supports online hot-swappable upgrades and has built-in device self-test functions.

[0012] Furthermore, the large-scale model task planning agent includes: The task planning agent is used to break down natural language flow instructions and autonomously arrange the execution sequence of Skills. Real-time control and coordination agent, used to arbitrate bus conflicts, balance bus load and manage global security interlocks; The operation and maintenance self-diagnosis agent is used to collect equipment operation data and perform fault root cause location and autonomous optimization of operating parameters through large model inference.

[0013] Furthermore, the AI ​​inference core and the real-time motion core are physically isolated by memory; the real-time motion core independently executes emergency stop cutoff, overtravel stop, overload protection, and dangerous output cutoff triggered by the communication watchdog; when the large model task planning agent is unavailable or communication is interrupted, the real-time execution and security module continues to run with the most recently verified execution graph, or switches to local threshold rule control.

[0014] A physical AI brain control method based on Agent and Skill architecture, applied to the aforementioned system, includes the following steps: S1. After the system is powered on, the EtherCAT real-time driver layer automatically scans all mounted slave devices and completes automatic network configuration and Skill plugin binding initialization. S2, the task planning agent receives the user's natural language flow instructions, breaks them down into multiple independent sub-tasks, and autonomously arranges the execution sequence and linkage logic of various Skill plugins; S3. After the production line triggers the workpiece arrival sensor signal, the sensor IO Skill uploads the trigger signal to the Agent layer. S4, Agent, schedules the visual skill to start CCD camera (including 2D or 3D camera) image acquisition, completes workpiece feature extraction, AI inference, target classification, precise positioning, size measurement or minor defect identification through built-in deep learning model, and outputs high-precision workpiece coordinate compensation data and quality inspection results. S5, the vision skill sends coordinate compensation data to the motion skill in real time through EtherCAT PDO, and controls and coordinates the agent to synchronize the servo motion cycle and vision acquisition cycle in real time. S6, the motion skill drives multi-axis servos to complete interpolation motion and point gripping actions, and works with sensor IO skill to complete clamping force and working condition detection; S7, the operation and maintenance self-diagnosis agent monitors the operating status of all devices and plug-ins in real time, corrects device operating parameters in real time, triggers hardware-level security interlocks under abnormal operating conditions, and completes fault alarms and autonomous source tracing.

[0015] Furthermore, when the constraint verification and execution graph compilation module verifies the control intent, it sequentially performs caller permission verification, Skill signature and version matching, hardware capability adaptation, security interlock check, resource conflict check, and verification that the sum of the worst execution times of all Skill nodes does not exceed the task deadline. Control intents that fail the verification only generate a rejection record and are archived by the audit and diagnosis module, without entering the real-time execution domain.

[0016] Furthermore, the image data from the CCD camera (including a 2D camera or a 3D camera) is transmitted to the AI ​​inference core via a GigE Vision or USB3Vision interface. The camera hard trigger signal and the workpiece arrival event are synchronously triggered by the EtherCAT slave station. The vision skill outputs the target position and confidence level with a timestamp. When the confidence level reaches a preset threshold and the safety interlock condition is met, the motion node is enabled; otherwise, a safety action is performed along the backtracking edge.

[0017] Furthermore, the communication core connects to the MES / ERP system via MQTTS and restricted APIs. External control commands are converted into the structured control intent after two-way authentication and permission verification, and the verification execution process of S2 to S7 is reused. The audit and diagnosis module associates and records the external command identifier with the corresponding execution graph version. The system is adapted to both industrial automation control scenarios and smart agriculture control scenarios.

[0018] The beneficial effects of this invention are as follows: This invention provides a physical AI brain control system and method based on an Agent and Skill architecture. By replacing the traditional PLC ladder diagram programming mode with an Agent and Skill industrial control architecture, it achieves production line logic configuration through natural language, significantly reducing the development and iteration costs of industrial automation and adapting to flexible intelligent manufacturing scenarios. It realizes the integrated integration of EtherCAT hard real-time bus, AI vision inference, and intelligent task scheduling, replacing multiple devices such as traditional PLCs, vision controllers, bus modules, and industrial control computers with a single device, resulting in high integration, low cost, and convenient deployment. Furthermore, through a flexible computing architecture using a general-purpose motherboard and an independent GPU, it is compatible with both ARM and Intel processors. The x86 dual-platform architecture, with GPU / NPU computing power dedicated to CCD vision deep learning inference and model fine-tuning tasks, allows for flexible hardware combinations based on production line computing power requirements, breaking free from the limitations of dedicated chip hardware and adapting to various industrial scenarios from high-end to low-end. Through built-in deep learning vision intelligent iteration capabilities, it can autonomously complete workpiece inspection, defect identification, and model parameter fine-tuning optimization. Combined with equipment self-diagnosis and adaptive control capabilities, it achieves intelligent quality inspection and maintenance on the production line. The Skill module supports secondary development and expansion, building dedicated Skill libraries for different equipment types. The same AI brain hardware can adapt to different physical equipment such as industrial robots, CNC machine tools, packaging equipment, agricultural irrigation equipment, and environmental control equipment, achieving truly universal intelligent brain functionality. The visual Skills simultaneously support traditional deep learning models and end-to-end large-scale visual algorithms (including large-scale visual models based on the Transformer architecture, large-scale visual-language models, or multimodal large-scale models), enabling… Advanced vision functions such as zero-sample defect detection, open-set target recognition, image semantic understanding, and visual question answering do not require pre-training models for specific scenarios, significantly improving the generalization ability and deployment efficiency of the vision system. Through a dual-scenario universal intelligent brain architecture, it is simultaneously adapted to two major fields: industrial automation control and smart agriculture intelligent control. It supports ECAT / CAN bus expansion of ARM microcontroller sub-control boards, forming a complete closed loop of large-scale model decision-making, main control scheduling, and local sub-control execution, refining underlying execution actions and providing extremely strong device scalability. A standardized Skill plugin ecosystem supports OTA remote upgrades, and with dual display interaction via touchscreen and external monitor, it boasts strong versatility and applicability. This invention abandons the traditional control mode of relying on code programming, program configuration, and fixed logic burning for industrial control equipment. It innovatively adopts a new equipment control architecture based on a large model, intelligent agent, and Skill library, enabling the definition, generation, and scheduling of equipment control logic through natural language dialogue, truly achieving zero-code intelligent equipment management. The core of the system relies on the AI ​​brain main control unit to build an intelligent control system, employing a fusion architecture combining a large model, intelligent agent, Skill library, and reinforcement learning.The large model, serving as the semantic decision-making core, is responsible for receiving user commands via text, voice, and other natural language dialogues. It accurately interprets user control intentions, operational scenario requirements, and parameter constraints, replacing the traditional manual development work of writing control processes and configuring technological logic. A dedicated intelligent agent for equipment control handles the core scheduling work, autonomously breaking down the large model's global commands into step-by-step executable equipment tasks, completing time-series planning and multi-device collaborative scheduling. Simultaneously, the system incorporates a reinforcement learning mechanism, enabling real-time collection of equipment operation data, operational effects, and environmental conditions. Through continuous trial-and-error iteration and reward optimization, it autonomously corrects its control strategies. The system pre-packages a standardized skill library for all industrial and agricultural scenarios, encapsulating hardware actions such as visual detection, motion positioning, water and fertilizer control, ambient lighting adjustment, equipment start / stop, and I / O linkage into independently callable standardized skill modules. Users can also add custom skills through secondary development to adapt to new equipment types and application scenarios. The agent, combining reinforcement learning optimization results, intelligently combines, invokes, and connects various skills to dynamically generate optimal device control logic adapted to the current scenario. The entire process requires no manual modification, compilation, or downloading of underlying control code, enabling autonomous iterative upgrades of the control logic and overcoming the limitations of traditional fixed programs. This solution completely overturns the traditional fixed-program control mode, relying on large-scale model semantic understanding, agent task scheduling, skill encapsulation, and reinforcement learning adaptive optimization capabilities. Users can manage device control, logic updates, and scene switching simply through chat. The system can autonomously adapt to complex working conditions and dynamically optimize control strategies, achieving code-free, flexible, and self-evolving intelligent control of equipment, significantly reducing the barriers to equipment development and maintenance. The visual skill supports 2D and 3D visual acquisition and processing, and also supports end-to-end large-scale model visual algorithms. Based on the continuous accumulation of world models, on-site operational data, and agents and skills, the system achieves adaptive evolution and cross-scenario migration of control strategies. The industrial brain's functions become increasingly comprehensive and universal with usage time. This invention balances autonomous decision-making by large models with the safety of hard real-time control, adapting to both industrial and agricultural scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only drawings of some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings. In addition, the drawings in the following description can be regarded as schematic diagrams and are not intended to limit the actual size of the product, the actual flow of the method, the actual timing of the signals, etc. involved in the embodiments of the present invention.

[0020] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0021] Figure 2 This is a schematic diagram of the overall layered architecture of the present invention.

[0022] Figure 3 This is a schematic diagram of the system of the present invention.

[0023] Figure 4 This is a schematic diagram of the autonomous control process of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages). In the description of this invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0026] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0027] like Figures 1 to 4 As shown, this embodiment of the invention provides a physical AI brain control system based on an Agent and Skill architecture, which includes a hardware layer and a software layer. The hardware layer comprises isolated AI inference cores, real-time motion cores, and communication cores, all encapsulated in an industrial dustproof and shockproof metal chassis. It supports a wide operating temperature range of -20℃ to 60℃, operates on 24V industrial DC power, and is compatible with DIN rail mounting. The AI ​​inference core is equipped with an NPU or GPU and runs large models for task planning and visual inference. The real-time motion core uses an Intel industrial motherboard or an ARM architecture industrial motherboard, equipped with an NPU or GPU to form the main control computing power AI brain hardware, running Windows, Linux, or HarmonyOS. It is used to perform multi-axis interpolation, electronic cams, and safety state machines, achieving 1ms-level motion control cycles and supporting advanced motion control functions such as multi-axis interpolation, electronic cams, synchronous following, homing, soft limit, and safety state machines. The communication core provides a CAN interface, a Gigabit Ethernet interface, and a USB 3.0 interface. The communication core integrates an RS485 Modbus-RTU, 4-20mA analog signal acquisition, thermocouple temperature measurement, Gigabit Ethernet, 5G industrial module, and WiFi 6. It also features an LVDS touch screen interface and an HDMI / DP display output interface, supporting local touch interaction on industrial touch screens and high-definition display on external monitors. The communication core also provides a USB 3.0 interface for high-speed CCD camera (including 2D and 3D cameras) access, and onboard EtherCAT and CAN bus expansion ports, allowing for the connection of multiple general-purpose ARM microcontrollers to control boards. The hierarchical execution architecture is clearly defined: the main control AI brain box is responsible for large model data analysis, intelligent judgment, automatic agent construction, global process scheduling, and deep learning visual reasoning (including 2D vision and 3D vision); the ARM sub-control board is located nearby to execute specific actions, including IO signal acquisition, sensor preprocessing, servo / stepping motion execution, solenoid valve switching, water and fertilizer pump control, fan and curtain adjustment, pulse output, local closed-loop interlocking and other low-level execution tasks, forming a distributed hierarchical control architecture with large model intelligent decision-making, main control global scheduling and sub-control precise local execution, which is fully adaptable to industrial multi-workstation clusters and smart agriculture distributed planting and breeding equipment clusters. The software layer includes: Device Digital Profiling Module, Skill Module, Large Model Task Planning Agent, Constraint Verification and Execution Graph Compilation Module, Real-time Execution and Security Module, Audit and Diagnosis Module, Security Rollback Module, and Continuous Learning and Evolution Module; The device digital profiling module is used to record device topology, slave station identifier, axis parameters, I / O points, camera calibration, available skills, and safety state machine. Among them, the device digital profiling module records slave station address, axis parameters, I / O points, camera calibration, device topology, hazardous areas, safety state machine, installed skills, compatible versions, and available bus resources. New hardware is scanned and authorized before being included in the profile. The Skill module is a standardized functional plugin. Each Skill declares its input / output type, resource usage, maximum execution time, trigger conditions, interlock conditions, rollback actions, and version number. The Skill module itself is a standardized functional plugin with an orchestration interface and real-time command segments. Each Skill declares its input / output, resource usage, real-time level, maximum execution time, trigger / interlock conditions, fault codes, rollback actions, version number, and signature. The Skill plugin layer modularly encapsulates four categories of standardized plugins. All plugins support hot-swapping, online upgrades, and independent version management. The interfaces are unified and standardized, including: unified JSON / binary PDO mapping of input / output parameters, support for online hot upgrades without interrupting device operation, built-in device self-detection functions and fault codes, and a bidirectional mapping channel between local API calls and bus data for the Agent layer to directly read and write device operating status. The Skill module supports adding custom Skill functions through secondary development, allowing users to customize functions for new devices. Develop dedicated skills for various equipment types (including but not limited to industrial robots, CNC machine tools, packaging equipment, agricultural irrigation equipment, environmental control equipment, etc.) and register them to the Skill module to achieve universal intelligent control of different physical devices using the same AI brain hardware. Motion skills include point-to-point control, multi-axis interpolation, electronic cams, synchronous following, torque limiting, homing, and soft limiters. Vision skills include a built-in deep learning inference engine with a lightweight industrial deep learning network model, providing functions such as object detection, semantic segmentation, feature comparison, sub-pixel localization, and minute defect detection. It supports the integration of 2D cameras (including ordinary industrial cameras) and 3D cameras (including structured light cameras, binocular cameras, and laser profilometers), and also supports the integration and inference of end-to-end large-scale visual algorithms, including large-scale visual models based on the Transformer architecture (such as ViT and DETR) and large-scale visual-language models (such as CLIP and Grounding). DINO) or multimodal large models (such as GPT-4V, LLaVA) can achieve zero-shot defect detection, open-set target recognition, image semantic understanding and visual question answering functions. There is no need to pre-train models for specific scenarios, which greatly improves the generalization ability and deployment efficiency of vision systems. It supports incremental learning of on-site samples and online fine-tuning and iteration of model parameters. The sensing I / OS skills include signal reading and writing, digital filtering, threshold alarm and high-speed counting. The agricultural skills include environmental acquisition, irrigation control, ventilation control, supplemental lighting control, valve interlock and pump protection. The large-scale model task planning agent receives natural language instructions or MES / API tasks, selects candidate skills based on the device's digital profile, and generates a structured control intent containing a set of candidate skills, deadlines, and fallback levels. Specifically, the model task planning agent receives natural language, MES / API tasks, or local events, selects candidate skills based on the digital profile, and generates control intents. These control intents include objectives, product formulations, candidate skill sets, priorities, allowed resources, deadlines, risk levels, and fallback levels. The agent layer comprises three main functional modules and relies on the top-level large-scale model to achieve automatic operational condition analysis, automatic agent generation, and dynamic iteration: the task planning agent parses natural language instructions and large-scale model analysis results, and automatically... It can dynamically decompose complex tasks, autonomously arrange the execution sequence and branch logic of Skills without manual programming, and support dynamic logic adaptation for loops, branches, and abnormal jumps; the real-time control and coordination agent unifies and synchronizes the bus clock, arbitrates conflicts between multiple devices, balances bus load, ensures the stability of hard real-time control, and provides global management and safety interlocks. When vision misses detection or sensor exceeds limits, it takes over globally and implements hardware-level safety interlocks to cut off motion output; the operation and maintenance self-diagnosis agent collects real-time operation data of all devices, including the running status of all Skill plugins, EtherCAT slave data, motor operating parameters, and camera working status. It infers the root cause of faults through industrial big data models, autonomously optimizes equipment operating parameters, achieves self-healing operation, and archives operation logs, pushes abnormal alarms and repair solutions. The constraint verification and execution graph compilation module sequentially performs equipment capability matching, safety interlock verification, resource conflict arbitration, and worst-case execution time verification on the control intent. Upon successful verification, it generates a real-time Skill execution graph with version identifier, trigger edge, resource lock, deadline, and rollback edge. Specifically, the constraint verification and execution graph compilation module verifies the following in sequence: caller identity and permissions, Skill signature and version, hardware capabilities, emergency stop / overtravel / fixture / danger zone interlocks, axis / camera / I / O / bandwidth resource conflicts, and worst-case execution time. Upon successful verification, it generates a real-time Skill execution graph with version number, trigger edge, resource lock, deadline, and rollback edge. Rejected intents only generate a reason record and are not processed on-site. The real-time execution and safety module executes the execution diagram at fixed intervals (0.25ms~10ms), driving slave devices through the EtherCAT master station and peripheral control boards through the CAN interface. Specifically, the real-time execution and safety module executes real-time instruction segments according to the execution diagram at fixed intervals, performing EtherCAT CoE PDO exchange, necessary SDO access, DC synchronization, and slave status monitoring. Conditions such as emergency stop, overtravel, overload, communication watchdog, low liquid level, and dangerous output cutoff are handled independently by the FPGA or real-time core. AI inference, cloud connectivity, or upper-layer API lag does not affect this safety path. The audit and diagnostic module records control intent, execution graph version, skill version, camera timestamp, axis position, slave status, and alarms. It publishes permission-filtered audit data through the MQTTS / API interface. Specifically, the audit and diagnostic module records control intent, execution graph version, skill version, camera timestamp, axis position, slave status, motor current, tracking error, frame rate, image quality, and alarms. The MQTTS / API only publishes permission-filtered data, and external write operations must be re-converted into control intents and follow the same verification chain. The safety rollback module configures trigger conditions such as timeout, insufficient confidence, interlock failure, device offline, bus abnormality, and agent unavailability for each execution graph and key skill. After triggering, it implements measures according to predetermined priorities, such as maintaining the last safe output, freezing approved execution graphs, stopping related axes, cutting off dangerous outputs, or switching to rule control, without using unverified temporary agent results. The continuous learning and evolution module is used to continuously optimize the Agent's decision-making logic and Skill execution strategy based on the world model, field operation data, and historical control records. The world model is used for unified modeling and simulation of environmental conditions, equipment characteristics, and process logic. Field operation data includes equipment status data, visual inspection results, motion execution feedback, environmental sensor data, and Agent decision logs. The system feeds back field operation data and Agent decision records to the world model for offline training and online fine-tuning, achieving adaptive iteration and cross-scenario generalization of control strategies. The Agent's experience knowledge and Skill library can be migrated and reused across different devices and scenarios. In new scenarios, there is no need to rewrite the control logic; adaptation can be achieved quickly simply by migrating the accumulated Agent experience and Skill library. As operating time increases, field data accumulates, and the Skill library continues to expand, the general control capabilities of the industrial brain are continuously enhanced, and the scope of functions is continuously expanded, achieving self-evolving intelligent control that becomes more comprehensive and universal with use.

[0028] The system constructs a closed-loop self-evolution mechanism based on a world model, field operation data, and the continuous accumulation of agents and skills. The world model is used for unified modeling and simulation of environmental states, equipment characteristics, and process logic. Field operation data includes equipment status data, visual inspection results, motion execution feedback, environmental sensor data, and agent decision logs. The system achieves adaptive iteration and cross-scenario generalization of control strategies by feeding field operation data and agent decision records back into the world model for offline training and online fine-tuning. In new scenarios, there is no need to rewrite the control logic; adaptation can be achieved quickly simply by migrating the accumulated agent experience and skill library. As operating time increases and the skill library continues to expand, the functional coverage of the industrial brain continuously expands, and its general control capabilities are continuously enhanced.

[0029] The EtherCAT real-time driver layer is based on secondary development of the IGH protocol stack, coupled with FPGA hardware acceleration. The DC distributed clock achieves a multi-slave synchronization error of less than 1μs. The memory isolation mechanism completely isolates the real-time motion process from the AI ​​inference process, so that the real-time performance and safety of motion control are not affected when the AI ​​is running under high load.

[0030] In one embodiment, the real-time motion core has a built-in dual-redundant EtherCAT RJ45 interface and uses a DC distributed clock to achieve microsecond-level synchronization of multiple slave stations, supporting various EtherCAT slave devices. These EtherCAT slave devices include motion slaves, IO sensor slaves, vision slaves, and auxiliary slaves. Motion slaves include EtherCAT servo drivers, stepper drivers, linear motors, and DD motors. IO sensor slaves include remote EtherCAT IO modules, pressure sensors, temperature sensors, and laser rangefinders. Vision slaves are GigE / USB3 Vision industrial CCD cameras (including 2D and 3D cameras), employing a dual-protocol linkage architecture of EtherCAT and GigE. Auxiliary slaves include valve islands, weighing modules, and barcode scanners. The AI ​​inference core's hardware platform is either an ARM industrial motherboard or an Intel x86 industrial motherboard, paired with a GPU computing module providing adjustable computing power from 6 to 50 TOPS. The system controller employs a dual-adaptive hardware solution: Solution 1 uses an ARM industrial motherboard with an embedded NPU / GPU, offering compact size and low power consumption, suitable for lightweight production lines and small-scale automated equipment; Solution 2 uses an Intel x86 industrial motherboard with a dedicated industrial GPU, providing powerful computing power and broad compatibility, suitable for multi-camera parallel visual inspection, multi-axis complex synchronous motion, and large-model agent scheduling scenarios. AI inference computing power is provided by the motherboard's GPU module. The ARM motherboard solution can be equipped with an onboard NPU or an embedded GPU, meeting a basic computing power of 6 TOPS, suitable for basic visual inspection and lightweight agent scheduling. The x86 motherboard solution can connect to a high-performance discrete GPU, with a computing power of 20-50 TOPS. It supports large model agent scheduling, parallel inference of multiple CCD cameras, and motion AI optimization. It is equipped with 128GB or more of industrial SSD storage for model libraries, plug-in libraries, and operation logs to meet the computing power requirements of different levels of industrial scenarios. The real-time motion core runs on Windows, Linux, or HarmonyOS, with a fixed cycle of 0.25ms to 10ms. Among them, the real-time motion core running on Windows, Linux, or HarmonyOS can stably achieve a 1ms motion control cycle and supports advanced motion control functions such as multi-axis interpolation, electronic cams, synchronous following, homing, soft limit, and safety state machines.

[0031] In one embodiment, the communication core is connected to multiple ARM microcontroller sub-control boards via EtherCAT or CAN bus; the main control AI brain is responsible for large model analysis, agent construction, global process scheduling and AI inference, and the ARM sub-control boards are used to perform high-speed IO acquisition, point motion, pulse output, sensor preprocessing, valve group start and stop and actuator closed-loop control locally.

[0032] In one embodiment, the Skill module includes a Motion Skill, a Vision Skill, a Sensing I / O Skill, and an Agricultural Skill. The Motion Skill includes point-to-point control, multi-axis interpolation, electronic cam, synchronous following, torque limiting, homing, and soft limit. The Vision Skill includes 2D vision acquisition, 3D vision acquisition, target positioning, size measurement, defect detection, calibration, and coordinate compensation, supporting the access of 2D cameras, 3D cameras (including structured light cameras, binocular cameras, and laser profilometers), and ordinary industrial cameras. The Sensing I / O Skill includes signal reading and writing, digital filtering, threshold alarm, and high-speed counting. The Agricultural Skill includes environmental acquisition, irrigation control, ventilation control, supplemental lighting control, valve interlocking, and pump protection. Each Skill supports online hot-swappable upgrades and has built-in device self-test functions.

[0033] In one embodiment, the large model task planning agent includes: The task planning agent is used to break down natural language flow instructions and autonomously arrange the execution sequence of Skills. Real-time control and coordination agent, used to arbitrate bus conflicts, balance bus load and manage global security interlocks; The operation and maintenance self-diagnosis agent is used to collect equipment operation data and perform fault root cause location and autonomous optimization of operating parameters through large model inference.

[0034] In one embodiment, the AI ​​inference core and the real-time motion core are physically isolated by memory; the FPGA hardware layer independently executes emergency stop cutoff, overtravel stop, overload protection, and dangerous output cutoff triggered by the communication watchdog; when the large model task planning agent is unavailable or communication is interrupted, the real-time execution and security module continues to run the execution graph that has been most recently verified, or switches to local threshold rule control.

[0035] This invention employs a dual hardware and software safety mechanism. The core safety interlocking logic is decentralized to the FPGA hardware layer. High-risk operating conditions such as emergency stop, overtravel, overload, and missed detection are directly cut off by the hardware, preventing equipment safety accidents caused by upper-layer AI model failure or program lag. Simultaneously, the hardware features a three-layer isolation design, independent power supply, and EMC industrial anti-interference design, making it adaptable to complex industrial environments. Hardware watchdog and power-off storage functions ensure data integrity and stable operation after abnormal restarts.

[0036] The safety rollback mechanism configures trigger conditions such as timeout, insufficient confidence, interlock failure, device offline, bus anomaly, and agent unavailability for each execution graph and critical skill. Upon triggering, it implements actions according to predetermined priorities, such as maintaining the last safe output, freezing approved execution graphs, stopping relevant axes, cutting off dangerous outputs, or switching to rule control, without using unverified temporary agent results.

[0037] A physical AI brain control method based on Agent and Skill architecture, applied to the above-mentioned system, includes the following steps: S1. After the system is powered on, the EtherCAT real-time driver layer automatically scans all mounted slave devices and completes automatic network configuration and Skill plugin binding initialization. S2, the task planning agent receives the user's natural language flow instructions, breaks them down into multiple independent sub-tasks, and autonomously arranges the execution sequence and linkage logic of various Skill plugins; S3. After the production line triggers the workpiece arrival sensor signal, the sensor IO Skill uploads the trigger signal to the Agent layer. S4, Agent, schedules the visual skill to start CCD camera (including 2D or 3D camera) image acquisition, completes workpiece feature extraction, AI inference, target classification, precise positioning, size calculation or minor defect identification through built-in deep learning model, and outputs high-precision workpiece coordinate compensation data (including two-dimensional coordinates and three-dimensional coordinates) and quality inspection results. S5, the vision skill sends coordinate compensation data to the motion skill in real time through EtherCAT PDO, and controls and coordinates the agent to synchronize the servo motion cycle and vision acquisition cycle in real time. S6, the motion skill drives multi-axis servos to complete interpolation motion and point gripping actions, and works with sensor IO skill to complete clamping force and working condition detection; S7, the operation and maintenance self-diagnosis agent monitors the operating status of all devices and plug-ins in real time, corrects device operating parameters in real time, triggers hardware-level security interlocks under abnormal operating conditions, and completes fault alarms and autonomous source tracing.

[0038] After the equipment is powered on and initialized, the system automatically scans the EtherCAT / CAN bus slave stations to complete the automatic network configuration and Skill plugin binding of cameras, servos, sensors, and ARM sub-control boards; the top-level large model automatically scans the topology of field equipment and environmental data, autonomously analyzes the current application scenario, and automatically builds a dedicated control agent and operating logic adapted to industrial production lines / smart agriculture; after the operator inputs natural language flow instructions, the agent automatically generates standardized control timing sequences.

[0039] In industrial settings, the system performs deep learning-based visual inspection of workpieces, coordinate calculation, and precise servo-driven grasping and sorting. During industrial vision sorting, a sensor skill generates an arrival event, the camera hard-triggers image acquisition, and the large-scale visual model outputs results with timestamps, coordinates (including 2D and 3D coordinates), pose, and confidence level. In the real-time domain, axis positions are acquired and coordinate compensation is performed using the same time reference; interpolation / grabbing is enabled only when confidence level, calibration status, fixture interlocking, and time budget are met. In cases of insufficient clamping force, visual missed detection, or slave station offline, the system executes image jump to re-image, defective product diversion, deceleration waiting, or safety shutdown.

[0040] In smart agriculture scenarios, crop growth and pest / disease status are visually identified, and combined with temperature, humidity, and soil sensor data, a control panel is activated to perform autonomous operations such as irrigation, fan temperature control, and curtain lighting. A CAN peripheral board or EtherCATI / O collects data on temperature, humidity, light intensity, and soil moisture content. The large-scale agent model can automatically generate candidate plans for irrigation, ventilation, and supplemental lighting; maximum continuous pump operation time, low-level protection, valve mutual exclusion, and basic threshold rules are implemented in the local execution domain.

[0041] When the network or agent is unavailable, the system retains the most recently approved security execution graph or switches to local rule control. The system can automatically fine-tune deep learning model parameters and agent scheduling logic based on newly added samples and operational data from the field, continuously improving detection accuracy and scenario adaptability.

[0042] In one embodiment, when the constraint verification and execution graph compilation module verifies the control intent, it sequentially performs caller permission verification, Skill signature and version matching, hardware capability adaptation, security interlock check, resource conflict check, and verification that the sum of the worst execution times of all Skill nodes does not exceed the task deadline. Control intents that fail the verification only generate a rejection record and are archived by the audit and diagnosis module, without entering the real-time execution domain.

[0043] In one embodiment, image data from a CCD camera (including a 2D camera or a 3D camera) is transmitted to the AI ​​inference core via a GigE Vision or USB3 Vision interface. The camera hard trigger signal and the workpiece arrival event are synchronously triggered by the EtherCAT slave station. The vision skill outputs the target position and confidence level with a timestamp. When the confidence level reaches a preset threshold and the safety interlock condition is met, the motion node is enabled; otherwise, a safety action is performed along the backtracking edge.

[0044] In one embodiment, the communication core connects to the MES / ERP system via MQTTS and restricted API. External control commands are converted into structured control intents after two-way authentication and permission verification, and the verification execution process from S2 to S7 is reused. The audit and diagnosis module associates and records the external command identifier with the corresponding execution graph version. The system is also compatible with industrial automation control scenarios and smart agriculture control scenarios.

[0045] In one specific embodiment, the AI ​​brain box is connected to an EtherCAT / CAN master station. The EtherCAT network connects to the interfaces of three-axis servos, remote I / O, photoelectric sensors, fixture valve islands, and encoders; CCD cameras (including 2D and / or 3D cameras) are connected to the AI ​​inference core via GigE Vision and hard-triggered via EtherCAT I / O. After the user or MES provides the product task, the large model agent automatically generates control intentions for position detection, image acquisition and positioning, coordinate compensation, multi-axis interpolation, clamping confirmation, and sorting output. After verifying that the sum of camera, axis, fixture, and conveyor line resources, as well as the worst-case execution time, does not exceed the task deadline, the system solidifies the execution diagram.

[0046] Compensation coordinates (including 2D and 3D coordinates) are issued and gripping is initiated only when the visual confidence level reaches a preset threshold; if the clamping force is insufficient, the process enters the defective product sorting stage. In the event of a servo alarm, overtravel, or slave station disconnection, the real-time motion core immediately stops the relevant axis and outputs safety I / O. The entire process stores the execution graph version, Skill version, image timestamp, and axis position for MES querying.

[0047] In another specific embodiment, the physical AI brain connects to multiple regional control boards via CAN. These boards control pumps, valves, fans, and supplemental lighting equipment, and collect data on temperature, humidity, light intensity, and soil moisture content. The large-scale model agent automatically generates candidate skill processes for irrigation, ventilation, and supplemental lighting based on natural language objectives or planting recipes. The constraint module checks pump current, low liquid level, valve mutual exclusion, and maximum continuous operating time; once these are satisfied, the local execution graph drives the peripheral boards.

[0048] When the cloud, MQTTS, or Agent is unavailable, the peripheral board continues to operate according to the approved execution graph or safety threshold rules; logs and alarms are uploaded after the network is restored. Therefore, this invention can replace the PLC main control function in small and medium-sized industrial equipment, and can also be used for smart agricultural control requiring secure autonomy.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A physical AI brain control system based on an Agent and Skill architecture, characterized in that, Includes hardware and software layers; The hardware layer includes an isolated AI inference core, a real-time motion core, and a communication core. The AI ​​inference core is equipped with an NPU or GPU and runs a large model for task planning and visual inference. The real-time motion core uses an Intel industrial motherboard or an ARM architecture industrial motherboard, equipped with an NPU or GPU to form the main control computing power AI brain hardware, and runs Windows, Linux, or HarmonyOS operating system for multi-axis interpolation, electronic cams, and safety state machines. The communication core provides a CAN interface, a Gigabit Ethernet interface, a USB3 Vision interface, and an MQTTS / API interface; The software layer includes: The equipment digital profiling module is used to record equipment topology, slave station identification, axis parameters, I / O points, camera calibration, available skills, and safety state machine; The Skill module is a standardized functional plugin. Each Skill declares the input / output type, resource consumption, maximum execution time, triggering condition, interlocking condition, rollback action, and version number. The large model task planning agent receives natural language instructions or MES / API tasks, selects candidate skills based on the device's digital profile, and generates a structured control intent that includes a set of candidate skills, a deadline, and a fallback level. The constraint verification and execution graph compilation module performs device capability matching, security interlock verification, resource conflict arbitration, and worst-case execution time verification on the control intent in sequence. After the verification is passed, a real-time Skill execution graph with version identifier, trigger edge, resource lock, deadline and rollback edge is generated. The real-time execution and safety module executes the execution diagram at a fixed cycle, drives the slave devices through the EtherCAT master station, and drives the peripheral control board through the CAN interface. The audit and diagnosis module is used to record the control intent, the execution graph version, the skill version, camera timestamp, axis position, slave status and alarms, and publish the permission-filtered audit data through the MQTTS / API interface.

2. The physical AI brain control system based on Agent and Skill architecture according to claim 1, characterized in that: The real-time motion core has a built-in dual redundant EtherCAT RJ45 interface and uses a DC distributed clock to achieve microsecond-level synchronization of multiple slave stations; the hardware platform of the AI ​​inference core is an ARM industrial motherboard or an Intel X86 industrial motherboard, and the matching GPU computing power module provides 6~50 TOPS of adjustable computing power; the real-time motion core runs Windows, Linux or HarmonyOS operating system, and the fixed period is 0.25ms~10ms.

3. The physical AI brain control system based on Agent and Skill architecture according to claim 1, characterized in that: The communication core is connected to multiple ARM microcontroller sub-control boards via EtherCAT or CAN bus; the main control AI brain is responsible for large model analysis, agent construction, global process scheduling and AI inference, and the ARM sub-control boards are used to perform high-speed IO acquisition, point movement, pulse output, sensor preprocessing, valve group start and stop and actuator closed-loop control locally.

4. The physical AI brain control system based on Agent and Skill architecture according to claim 1, characterized in that: The Skill module includes Motion Skill, Vision Skill, Sensor I / O Skill, and Agriculture Skill; the Motion Skill includes point-to-point control, multi-axis interpolation, electronic cam, synchronous following, torque limiting, homing, and soft limit; the Vision Skill includes 2D vision acquisition, 3D vision acquisition, target positioning, size measurement, defect detection, calibration, and coordinate compensation, supporting the connection of 2D cameras, 3D cameras, and ordinary industrial cameras; the Sensor I / O Skill includes signal reading and writing, digital filtering, threshold alarm, and high-speed counting; the Agriculture Skill includes environmental acquisition, irrigation control, ventilation control, supplemental lighting control, valve interlocking, and pump protection; each Skill supports online hot-swappable upgrades and has built-in device self-test functions.

5. A physical AI brain control system based on an Agent and Skill architecture as described in claim 1, characterized in that: The large-scale model task planning agent includes: The task planning agent is used to break down natural language flow instructions and autonomously arrange the execution sequence of Skills. Real-time control and coordination agent, used to arbitrate bus conflicts, balance bus load and manage global security interlocks; The operation and maintenance self-diagnosis agent is used to collect equipment operation data and perform fault root cause location and autonomous optimization of operating parameters through large model inference.

6. A physical AI brain control system based on an Agent and Skill architecture as described in claim 1, characterized in that: The AI ​​inference core and the real-time motion core are physically isolated by memory. The real-time motion core independently executes emergency stop cutoff, overtravel stop, overload protection, and dangerous output cutoff triggered by the communication watchdog. When the large model task planning agent is unavailable or communication is interrupted, the real-time execution and security module continues to run the execution graph that has passed the most recent verification, or switches to local threshold rule control.

7. A physical AI brain control method based on an Agent and Skill architecture, applied to the system described in any one of claims 1-6, characterized in that: Includes the following steps: S1. After the system is powered on, the EtherCAT real-time driver layer automatically scans all mounted slave devices and completes automatic network configuration and Skill plugin binding initialization. S2, the task planning agent receives the user's natural language flow instructions, breaks them down into multiple independent sub-tasks, and autonomously arranges the execution sequence and linkage logic of various Skill plugins; S3. After the production line triggers the workpiece arrival sensor signal, the sensor IO Skill uploads the trigger signal to the Agent layer. S4, Agent, schedules the visual skill to start CCD camera image acquisition, and completes workpiece feature extraction, AI inference, target classification, precise positioning, size measurement or minor defect identification through the built-in deep learning model, and outputs high-precision workpiece coordinate compensation data and quality inspection results. S5, the vision skill sends coordinate compensation data to the motion skill in real time through EtherCAT PDO, and controls and coordinates the agent to synchronize the servo motion cycle and vision acquisition cycle in real time. S6, the motion skill drives multi-axis servos to complete interpolation motion and point gripping actions, and works with sensor IO skill to complete clamping force and working condition detection; S7, the operation and maintenance self-diagnosis agent monitors the operating status of all devices and plug-ins in real time, corrects device operating parameters in real time, triggers hardware-level security interlocks under abnormal operating conditions, and completes fault alarms and autonomous source tracing.

8. The physical AI brain control method based on Agent and Skill architecture according to claim 7, characterized in that: When the constraint verification and execution graph compilation module verifies the control intent, it sequentially performs caller permission verification, Skill signature and version matching, hardware capability adaptation, security interlock check, resource conflict check, and verification that the sum of the worst execution times of all Skill nodes does not exceed the task deadline. Control intents that fail validation only generate a rejection record and are archived by the audit and diagnostic module, without entering the real-time execution domain.

9. A physical AI brain control method based on Agent and Skill architecture according to claim 7, characterized in that: The image data from the CCD camera is transmitted to the AI ​​inference core via a GigE Vision or USB3 Vision interface, and the camera hard trigger signal and workpiece arrival event are synchronously triggered by the EtherCAT slave station. The visual skill outputs the target position and confidence level with a timestamp. When the confidence level reaches a preset threshold and the safety interlock condition is met, the motion node is enabled; otherwise, a safety action is performed along the backtrack edge.

10. A physical AI brain control method based on Agent and Skill architecture according to claim 7, characterized in that: The communication core connects to the MES / ERP system via MQTTS and restricted APIs. External control commands are converted into the structured control intent after two-way authentication and permission verification, and the verification execution process of S2 to S7 is reused. The audit and diagnosis module associates and records the external command identifier with the corresponding execution graph version. The system is adapted to both industrial automation control scenarios and smart agriculture control scenarios.

Images