An unmanned equipment training-free adaptive control method based on knowledge enhancement and quality control

Abstract

The application discloses a kind of unmanned equipment training-free self-adaptive control methods based on knowledge enhancement and quality control.The method collects the visual input of heterogeneous unmanned equipment entering new operation environment, task instruction and equipment state information, generates scene data to be adapted;The scene data to be adapted is input into the edge side fixed weight lightweight VLM, and the environment feature description is generated;Based on the environment feature description, the prompt word template and the environment constraint rule are dynamically retrieved from the external scene prompt word library and the environment knowledge base, and the knowledge enhancement prompt information is formed;The knowledge enhancement prompt information is input into the fixed weight lightweight VLM to generate structured control intent;Based on the equipment capability dictionary, the control strategy is generated and quality control is checked;The control strategy that passes the check is converted into bottom control instruction execution, and is dynamically revised based on feedback, realizes training-free scene self-adaptive control under the condition of fixed model weight.Can improve the scene adaptation capability and control safety of unmanned equipment.

Description

Technical Field

[0001] This invention relates to the fields of intelligent control of unmanned equipment, edge intelligence, multimodal perception, visual language models, and retrieval enhancement generation technology, and particularly to a training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control. Background Technology

[0002] With the increasing demand for unmanned operations in scenarios such as mining, military industry, emergency rescue, and industrial inspection, heterogeneous unmanned equipment such as unmanned vehicles, drones, tracked robots, and inspection robots are widely used for tasks such as target recognition, path movement, area inspection, and anomaly handling. Existing unmanned equipment typically relies on pre-trained models, scene recognition models, or task control models to complete environmental understanding and action execution, which can meet basic usage requirements in fixed scenarios.

[0003] However, actual operating environments such as mines and military facilities are highly uncertain and variable. The terrain, lighting conditions, obstacle distribution, hazardous areas, mission rules, and safety constraints may all differ at different sites. When unmanned equipment enters a new environment, existing solutions often require re-collecting data and fine-tuning or updating the model, resulting in long deployment cycles, high delivery costs, and difficulty in meeting the needs for rapid deployment and large-scale replication.

[0004] While visual language models can enhance unmanned equipment's understanding of images, text, and task commands, their direct application in control still suffers from issues such as insufficient scene adaptation, unstable output, ambiguous action mapping, and inadequate safety and quality control. Therefore, an adaptive control method for unmanned equipment is needed that, under conditions of fixed model weights, no retraining, and no on-site fine-tuning, achieves environment understanding switching, action mapping adaptation, and a closed-loop quality control through a scene cue lexicon, environmental knowledge base, and retrieval enhancement mechanisms. Summary of the Invention

[0005] This invention provides a training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control, comprising: S10. Collect visual input, task instructions and equipment status information of heterogeneous unmanned equipment after entering a new working environment, and integrate them to generate scene data to be adapted, which represents the semantic structure of the current scene, task objectives and equipment execution constraints. S20. Input the scene data to be adapted into a lightweight VLM deployed on the edge side with fixed model weights, so that the lightweight VLM is only used for scene semantic parsing and does not participate in model parameter updates or fine-tuning training, and generates an environmental feature description for triggering control logic selection. S30. Based on the description of environmental features, the system dynamically retrieves and matches prompt word templates and environmental constraint rules from the external scene prompt word library and environment knowledge base through a retrieval enhancement mechanism, and uses the prompt word templates as the constraint source for the control logic generation structure to form knowledge-enhanced prompt information. S40. Input the knowledge-enhanced prompt information as the control generation constraint into the fixed-weight lightweight VLM, so that the lightweight VLM can generate structured control intent based on the control structure defined by the prompt word template without retraining. S50. Based on the structured control intent and the heterogeneous unmanned equipment capability dictionary, establish the mapping relationship between action semantics and equipment control interface to generate control strategy. Before execution, perform task consistency constraint verification, safety constraint verification and equipment executability verification on the control strategy. Only when the verification constraints are met is the execution phase allowed. S60. The control strategy that has passed the quality control verification is converted into the underlying control command of the unmanned equipment and issued for execution. Based on the execution feedback, the environmental feature description and prompt word retrieval results are dynamically updated to trigger the reconstruction of control intent and the correction of control strategy, forming a training-free adaptive control closed loop based on fixed model weights.

[0006] The above-described knowledge-enhanced and quality-controlled training-free adaptive control method for unmanned equipment involves collecting visual input, task commands, and equipment status information of heterogeneous unmanned equipment after entering a new operating environment, and fusing them to generate adaptive scene data representing the semantic structure of the current scene, task objectives, and equipment execution constraints. This includes: S101. Collect visual images, video streams and environmental spatial structure information after the heterogeneous unmanned equipment enters the new working environment, and identify target objects, scene boundaries, obstacle distribution and passable areas in the current environment to form a basic semantic representation of the scene. S102. Synchronously collect task instructions, equipment motion status, load status, communication status and equipment execution constraint parameters, and integrate task objectives, equipment execution conditions and scene semantic basic representation to generate scene data to be adapted.

[0007] The above-described knowledge-enhanced and quality-controlled unmanned equipment training-free adaptive control method involves inputting the scene data to be adapted into a lightweight Virtual Machine (VLM) deployed at the edge with fixed model weights. This lightweight VLM is used solely for scene semantic parsing and does not participate in model parameter updates or fine-tuning training, generating environmental feature descriptions to trigger control logic selection, including: S201. Input the scene data to be adapted into the lightweight VLM deployed on the edge side. The lightweight VLM performs multimodal semantic parsing on the visual input, task instructions and equipment status information, and keeps the model weights unchanged during the on-site deployment process. S202. Extract scene type, target object, obstacle distribution, dangerous area, passable area and equipment status based on the multimodal semantic parsing results, and generate an environmental feature description to trigger subsequent prompt word retrieval and control logic selection.

[0008] The above-described knowledge-enhanced and quality-controlled unmanned equipment training-free adaptive control method, wherein, based on environmental feature description, a retrieval enhancement mechanism dynamically retrieves matching prompt word templates and environmental constraint rules from an external scene prompt word library and an environmental knowledge base, and uses the prompt word templates as the constraint source for the control logic generation structure to form knowledge-enhanced prompt information, including: S301. Based on the environmental feature description, perform semantic retrieval of the external scene prompt word library and match prompt word templates corresponding to the current scene type, target object, task target and device execution conditions; S302. Retrieve the operation rules, safety constraints, action boundaries, abnormal handling rules and result verification requirements corresponding to the current scenario from the environmental knowledge base, and combine them with the matched prompt word template to form knowledge-enhanced prompt information.

[0009] The above-described knowledge-enhanced and quality-controlled unmanned equipment training-free adaptive control method, wherein knowledge-enhanced prompts are used as control generation constraints input to a fixed-weight lightweight VLM, enabling the lightweight VLM to generate structured control intentions based on a control structure defined by a prompt word template without retraining, including: S401. Input the knowledge-enhanced prompts, environmental feature descriptions, and task instructions into the fixed-weight lightweight VLM. Use prompt word templates to constrain the scene understanding method and control inference path of the lightweight VLM to trigger the switching of control logic structure under different scenarios. S402. A structured control intent is generated by a fixed-weight lightweight VLM without model parameter updates or on-site fine-tuning. The structured control intent includes the target object, target action, action path, safety boundary, prohibited actions, anomaly handling rules, and verification requirements.

[0010] The above-described knowledge-enhanced and quality-controlled unmanned equipment training-free adaptive control method, wherein a control strategy is generated by establishing a mapping relationship between action semantics and equipment control interfaces based on structured control intentions and heterogeneous unmanned equipment capability dictionaries, and performs task consistency constraint verification, safety constraint verification, and equipment executability verification on the control strategy before execution, including: S501. Based on the structured control intent, call the heterogeneous unmanned equipment capability dictionary to determine the motion capability, perception capability, execution capability, communication capability and underlying control interface type supported by the current unmanned equipment, and establish the semantic mapping relationship between the target action and the underlying control interface. S502. Generate a control strategy based on the semantic mapping relationship, and perform task consistency verification, security constraint verification, device executability verification, and result verifiability verification on the control strategy. Only when the control strategy meets the verification constraints is the execution phase allowed.

[0011] The above-described knowledge-enhanced and quality-controlled unmanned equipment training-free adaptive control method, wherein the control strategy verified by quality control is converted into the underlying control commands of the unmanned equipment and executed, and the environmental feature description and prompt word retrieval results are dynamically updated based on the execution feedback to trigger control intent reconstruction and control strategy correction, forming a training-free adaptive control closed loop based on fixed model weights, including: S601. Convert the control strategy that has passed the quality control verification into motion control commands, attitude control commands, task execution commands, abnormal stop commands or rollback control commands that can be recognized by the underlying control interface of the unmanned equipment, and send them to the corresponding unmanned equipment for execution. S602. Collect visual feedback, position feedback, attitude feedback, equipment status feedback and task result feedback during the execution of unmanned equipment, and update the environmental feature description and prompt word retrieval results based on the feedback results to trigger control intent reconstruction and control strategy correction, forming a training-free adaptive control closed loop.

[0012] This invention also provides a training-free adaptive control system for unmanned equipment based on knowledge enhancement and quality control, comprising: The scene data acquisition module is used to collect visual input, task instructions and equipment status information after heterogeneous unmanned equipment enters a new operating environment, and integrate them to generate scene data to be adapted. The environment feature generation module is used to input the scene data to be adapted into a lightweight VLM deployed on the edge with fixed model weights, and generate an environment feature description for triggering the selection of control logic. The knowledge-enhanced retrieval module is used to dynamically retrieve matching prompt word templates and environmental constraint rules from the plug-in scene prompt word library and environment knowledge base based on environmental feature descriptions, and form knowledge-enhanced prompt information; The structured control intent generation module is used to input knowledge-enhanced prompts as control generation constraints into a fixed-weight lightweight VLM to generate structured control intents. The control strategy generation and quality control module is used to generate control strategies by establishing a mapping relationship between action semantics and equipment control interfaces based on structured control intentions and heterogeneous unmanned equipment capability dictionaries, and to perform quality control verification on the control strategies in terms of task consistency, safety constraints, equipment executability and result verifiability. The execution and closed-loop feedback module is used to convert the control strategy that has passed the quality control verification into the underlying control command of the unmanned equipment and issue it for execution. Based on the execution feedback, it dynamically corrects the environmental feature description, prompt word retrieval results, control intention and control strategy to form a training-free adaptive control closed loop based on fixed model weights.

[0013] The beneficial effects achieved by this invention are as follows: By using a fixed-weight lightweight VLM, an external scene prompt word library, an environmental knowledge base, and a retrieval enhancement mechanism, this invention enables unmanned equipment to automatically match prompt word templates, scene rules, and action constraints based on visual input and environmental features after entering a new environment without retraining or on-site fine-tuning. This generates a structured control intent adapted to the current task and completes action mapping and control strategy generation in conjunction with the equipment capability dictionary. At the same time, through quality control verification and execution feedback closed loop, the control strategy is modified for safety, executability, and result verification, thereby improving the rapid deployment capability, standardized delivery capability, control safety, and scene adaptability of heterogeneous unmanned equipment in complex scenarios such as mines and military industries. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a flowchart of a training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control, provided in Embodiment 1 of this application.

[0016] Figure 2 This is a schematic diagram of a training-free adaptive control system for unmanned equipment based on knowledge enhancement and quality control, provided in Embodiment 2 of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 like Figure 1 As shown, Embodiment 1 of this application provides a training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control, including the following steps: S10. Collect visual input, task instructions and equipment status information of heterogeneous unmanned equipment after entering a new operating environment, and integrate them to generate scene data to be adapted, which represents the semantic structure of the current scene, task objectives and equipment execution constraints.

[0019] After heterogeneous unmanned equipment enters a new operating environment, the edge control terminal collects the current environmental visual input, task instructions, and equipment status information, and organizes the scene objects, task objectives, equipment capabilities, and safety constraints into scene data to be adapted, including the following sub-steps: S101. Collect visual images, video streams and environmental spatial structure information after the heterogeneous unmanned equipment enters the new working environment, and identify target objects, scene boundaries, obstacle distribution and passable areas in the current environment to form a basic semantic representation of the scene. Unmanned equipment collects visual images, video streams, and environmental spatial structure information in the new operating environment through forward-looking cameras, surround-view cameras, depth cameras, and lidar. It also performs key frame filtering on continuous video frames and aligns the depth point cloud with the image coordinates to obtain environmental perception data at the same time and with the same coordinate reference.

[0020] The edge control terminal identifies key objects and areas related to scene adaptation from environmental perception data, and records the category, location, distance, movement trend and boundary coordinates of task target objects, personnel, vehicles, water accumulation, falling rocks, alleyway boundaries, fence boundaries, restricted access boundaries, dangerous area boundaries and passable areas.

[0021] The edge control terminal generates a basic semantic representation of the scene based on the recognition results. The target object field records the task target and surrounding dynamic targets, the boundary field records the alley boundary, fence boundary and restricted area boundary, the obstacle field records the obstacle type, distance and occupied area, the passable area field records the passable direction and stop location, and the area risk field records risk semantics such as the target being adjacent to the restricted area boundary, the passable path being blocked by water accumulation, personnel entering the passable path, and the boundary with the risk of collapse on the right.

[0022] S102. Synchronously collect task instructions, equipment motion status, load status, communication status and equipment execution constraint parameters, and integrate task objectives, equipment execution conditions and scene semantic basic representation to generate scene data to be adapted.

[0023] The edge control unit synchronously collects task commands, equipment motion status, load status, communication status, and equipment execution constraint parameters. Task commands are either natural language commands or structured commands issued by the scheduling system; equipment motion status records position, speed, heading angle, attitude angle, and braking status; equipment execution constraint parameters record maximum travel speed, minimum turning radius, braking distance, obstacle crossing ability, hill climbing ability, robotic arm range of motion, sensor field of view, and action types supported by the underlying control interface.

[0024] The edge control unit binds the task objectives in the task instructions with the basic semantic representation of the scene, and generates execution constraint fields by combining the device execution constraint parameters. For example, if the task instruction is "inspect along the alleyway", and the basic semantic representation of the scene contains "personnel entering the travel path" and "water blocking the passage", the execution constraint field records personnel avoiding, maintaining a safe distance, slowing down, and detouring around the waterlogged area; if the task instruction is "approach the target device to take pictures", and the basic semantic representation of the scene contains "target is near a restricted boundary", the execution constraint field records prohibiting crossing the boundary, stopping outside the boundary, adjusting the shooting posture, and maintaining a safe distance.

[0025] The data for the scenarios to be adapted is organized according to scenario type, target object, task objective, boundary, obstacle, passable area, area risk, equipment status, equipment capability, scenario rule identifier, and execution constraint. The scenario rule identifier records the rules for passing through mine roadways, rules for military restricted areas, rules for avoiding dangerous areas, or rules for equipment shooting operations. The execution constraint records the speed, distance, boundary, attitude, and action restrictions that need to be met when the current task is executed.

[0026] S20. Input the scene data to be adapted into a lightweight VLM deployed on the edge with fixed model weights, so that the lightweight VLM is only used for scene semantic parsing and does not participate in model parameter updates or fine-tuning training, and generates an environmental feature description for triggering control logic selection.

[0027] Visual Language Model (VLM) is a visual language model capable of joint semantic parsing of images, video frames, and text commands. The edge control unit inputs the scene data to be adapted into a lightweight VLM deployed at the edge. With fixed model weights, the lightweight VLM parses the current scene, task objective, and equipment status to generate an environmental feature description, including the following sub-steps: S201. Input the scene data to be adapted into the lightweight VLM deployed on the edge side. The lightweight VLM performs multimodal semantic parsing on the visual input, task instructions and equipment status information, and keeps the model weights unchanged during the on-site deployment process. The edge control unit organizes the scene data to be adapted into a lightweight VLM-recognizable multimodal input. The multimodal input consists of a visual frame group and a semantic field group. The visual frame group includes the current environment image and video keyframes, and the semantic field group includes task instructions, basic semantic representation of the scene, regional risk field, equipment status field, equipment capability field, and execution constraint field.

[0028] The lightweight VLM performs semantic parsing on multimodal inputs, mapping targets, boundaries, obstacles, and passable areas in the visual image to task instructions and identifying constraints during task execution. For example, if the task instruction is "inspect along the alleyway" and the semantic field group contains "personnel entering the travel path" or "water blocking the passage," the lightweight VLM will parse the current task as an inspection task requiring personnel to avoid, slow down, and detour. If the task instruction is "approach the target device to take a picture" and the semantic field group contains "target is near a restricted boundary," the lightweight VLM will parse the current task as a shooting task requiring stopping outside the boundary and adjusting posture.

[0029] Lightweight VLM does not update model parameters, backpropagate gradients, or fine-tune training during field deployment, and the model weights remain fixed; adaptation information for different scenarios is provided by the scenario rule identifier, regional risk field, and execution constraint field in the data of the scenario to be adapted.

[0030] S202. Extract scene type, target object, obstacle distribution, dangerous area, passable area and equipment status based on the multimodal semantic parsing results, and generate an environmental feature description to trigger subsequent prompt word retrieval and control logic selection.

[0031] The edge control terminal generates an environmental feature description based on the semantic parsing results of the lightweight VLM. The environmental feature description is organized according to scene type, target object, obstacle distribution, hazardous area, passable area, device status, risk label, and action constraints.

[0032] The scene type records mine tunnels, military control areas, industrial inspection channels, or emergency rescue sites; the target object records the mission target and its spatial location; the obstacle distribution records the obstacle type, distance, occupied area, and movement trend; the dangerous area records water accumulation, landslides, restricted areas, high temperature areas, electrified areas, or other restricted areas; the passable area records the current travel direction, turning space, and docking location; and the equipment status records speed, attitude, power, communication, load, and low-level control interface status.

[0033] Environmental feature descriptions are output in the form of combined tags that can trigger searches, such as "mine roadway—personnel approaching—water accumulation blocking—slow down and detour—low-speed feasible", or "military industrial control area—target near restricted boundary—parking outside the boundary—attitude adjustment—communication normal". These combined tags, as the output form of environmental feature descriptions, correspond to the scene tags, risk tags, action tags, and equipment status tags in the prompt word template.

[0034] S30. Based on the environmental feature description, the system dynamically retrieves and matches prompt word templates and environmental constraint rules from the external scene prompt word library and environment knowledge base through a retrieval enhancement mechanism, and uses the prompt word templates as the constraint source for the control logic generation structure to form knowledge-enhanced prompt information.

[0035] The edge control terminal retrieves corresponding prompt word templates and environmental constraint rules from the external scene prompt word library and environment knowledge base based on the scene type, risk label, action constraints, and device status in the environmental feature description, and combines them to form knowledge-enhanced prompt information, including the following sub-steps: S301. Based on the environmental feature description, perform semantic retrieval of the external scene prompt word library and match prompt word templates corresponding to the current scene type, target object, task target and device execution conditions; The external scenario prompt word library organizes prompt word templates according to scenario tags, risk tags, action tags, and device status tags. Each prompt word template records the key points of environmental understanding, action generation structure, and verification requirements for a specific scenario. The prompt word template includes a scenario recognition section, a target confirmation section, an action generation section, a security restriction section, and a result verification section.

[0036] The edge control terminal uses the combined tags in the environmental feature description as search conditions to perform semantic retrieval on the external scene prompt word library, and determines the prompt word template based on the degree of matching between scene type, target object, task objective, risk tag, action constraint, and device status. For example, when the environmental feature description is "mine roadway - personnel approaching - water blockage - slow down and detour - low speed feasible", the prompt word template for low speed inspection in mine roadway is matched; when the environmental feature description is "military industrial control area - target near restricted boundary - stop outside the boundary - attitude adjustment - communication normal", the prompt word template for shooting outside the military industrial boundary is matched.

[0037] The matched prompt word templates are not used to change the model weights of the lightweight VLM, but rather to limit the control logic generation structure of the lightweight VLM in the current scenario. For example, the prompt word templates specify the target objects that need to be identified first, the risk areas that need to be avoided, the types of actions that can be generated, the types of actions that cannot be generated, and the verification method after the task is completed.

[0038] S302. Retrieve the operation rules, safety constraints, action boundaries, abnormal handling rules and result verification requirements corresponding to the current scenario from the environmental knowledge base, and combine them with the matched prompt word template to form knowledge-enhanced prompt information.

[0039] The environmental knowledge base stores environmental constraint rules according to the operation scenario, equipment type, and risk type, including rules for passing through mine roadways, rules for military restricted areas, rules for avoiding dangerous areas, rules for personnel safety distance, rules for low-speed passage, rules for parking outside the boundary, rules for robotic arm movement boundaries, rules for abnormal stopping, and rules for verifying task completion.

[0040] The edge control terminal calls the corresponding environmental constraint rules based on the matched prompt word template and writes the rule content into the corresponding paragraph of the prompt word template. For example, the prompt word template for low-speed inspection in mine roadways includes rules for personnel avoidance, water accumulation detour, roadway boundary maintenance, and collapse risk stop; the prompt word template for shooting outside military industrial boundaries includes rules for prohibited entry boundaries, shooting distance, parking outside boundaries, and boundary crossing trigger stop.

[0041] The combined knowledge-enhanced prompts are organized according to scenario understanding prompts, target recognition prompts, action constraint prompts, prohibited action prompts, anomaly handling prompts, and result verification prompts. Among them, action constraint prompts record the range of actions that can be performed, prohibited action prompts record that actions must not be taken beyond the boundary, must not approach people at high speed, or must not enter dangerous areas, anomaly handling prompts record the conditions for stopping, reverting, or reporting, and result verification prompts record the conditions for image acquisition, location arrival, distance maintenance, or anomaly reporting.

[0042] S40. Input the knowledge-enhanced prompt information as the control generation constraint into the fixed-weight lightweight VLM, so that the lightweight VLM can generate structured control intent based on the control structure defined by the prompt word template without retraining.

[0043] The edge control terminal inputs knowledge-enhanced prompts, environmental feature descriptions, and task instructions into a fixed-weight lightweight VLM, and generates structured control intents according to the field order in the prompt word template, including the following sub-steps: S401. Input knowledge-enhanced prompts, environmental feature descriptions, and task instructions into a fixed-weight lightweight VLM. Use prompt word templates to constrain the scene understanding method and control inference path of the lightweight VLM to trigger the switching of control logic structure in different scenarios. The edge control terminal writes the scene type, risk label, action constraints, and device status from the environmental feature description into the corresponding fields of the prompt word template, forming the input for generating control intent. The fields in the prompt word template include scene recognition field, target confirmation field, risk judgment field, permitted action field, prohibited action field, abnormal response field, and verification field.

[0044] When the environmental characteristics are described as mine roadway, personnel approaching, water accumulation blocking, and low-speed feasibility, the action fields are allowed to include low-speed advance, stop and wait, detour, and report, while the action fields are prohibited to include high-speed advance, approaching personnel, entering water accumulation areas, and crossing landslide boundaries. When the environmental characteristics are described as military industrial control area, target near restricted access boundary, and normal communication, the action fields are allowed to include parking outside the boundary, gimbal turning, attitude fine-tuning, and shooting, while the action fields are prohibited to include crossing restricted access boundary, getting close to sensitive equipment, and leaving the communication coverage area.

[0045] The fixed-weight lightweight VLM generates the control logic structure corresponding to the input and output scenarios based on the above control intent, and the model weights remain unchanged during on-site deployment.

[0046] S402. A structured control intent is generated by a fixed-weight lightweight VLM without model parameter updates or on-site fine-tuning. The structured control intent includes the target object, target action, action path, safety boundary, prohibited actions, exception handling rules, and verification requirements.

[0047] The fixed-weight lightweight VLM outputs structured control intents according to the target object, target action, action path, security boundary, prohibited actions, exception handling rules, and verification requirements. The fields are associated with the control logic in the same scenario.

[0048] In a mine roadway inspection scenario, the structured control intent is recorded as follows: the target objects are the inspection point ahead and personnel within the travel path; the target action is to move forward at low speed and avoid personnel; the action path is to bypass the water accumulation along the passable area on the left side of the roadway centerline; the safety boundary is to maintain a safe distance between personnel and avoid the collapse warning boundary; prohibited actions are to move forward at high speed, enter the water accumulation area, and approach the collapse boundary; the abnormal handling rule is to stop and report when the passable area disappears or personnel continue to occupy the path; the verification requirement is to reach the inspection point without entering the danger zone.

[0049] In shooting scenarios outside military industrial boundaries, the structured control intent is recorded as follows: the target object is the designated equipment; the target action is to approach and shoot; the action path is to move to a dockable position outside the restricted boundary and adjust the gimbal angle; the safety boundary is not to cross the restricted boundary and to maintain a limited shooting distance; prohibited actions are to cross the boundary, get close to sensitive equipment, and enter areas with weak communication coverage; the anomaly handling rule is to stop when the positioning drift or the risk of crossing the boundary increases; the verification requirement is to complete the acquisition of the target image and maintain the docked state outside the boundary.

[0050] S50. Based on the structured control intent and the heterogeneous unmanned equipment capability dictionary, establish the mapping relationship between action semantics and equipment control interface to generate control strategy, and perform task consistency constraint verification, safety constraint verification and equipment executability verification of control strategy before execution.

[0051] The edge control terminal calls the heterogeneous unmanned equipment capability dictionary according to the structured control intent, converts the target action into the control strategy supported by the current unmanned equipment, and performs quality control verification before issuing the strategy, including the following sub-steps: S501. Based on the structured control intent, call the heterogeneous unmanned equipment capability dictionary to determine the motion capability, perception capability, execution capability, communication capability and underlying control interface type supported by the current unmanned equipment, and establish the semantic mapping relationship between the target action and the underlying control interface. The heterogeneous unmanned equipment capability dictionary records the current equipment capabilities, interface types, and parameter boundaries of the unmanned equipment. The edge control terminal matches the corresponding underlying control interface and execution parameters based on the target action, action path, safety boundary, and anomaly handling rules in the structured control intent.

[0052] For example, the structured control intent "progress at low speed and avoid water accumulation" is mapped to a speed control interface, a steering control interface, and a path following interface, generating speed limits, detour directions, and obstacle avoidance distances; "stop outside the boundary and adjust the gimbal for shooting" is mapped to a mobile docking interface, a gimbal control interface, and an image acquisition interface, generating docking coordinates, gimbal angles, and number of acquisitions; "stop when boundary crossing risk increases" is mapped to an abnormal stop interface and an alarm reporting interface.

[0053] The semantic mapping relationship is generated in the form of "target action - control interface - execution parameters - boundary conditions". The execution parameters in the semantic mapping relationship are limited by the parameter boundaries in the device capability dictionary.

[0054] S502. Generate a control strategy based on the semantic mapping relationship, and perform task consistency verification, security constraint verification, device executability verification, and result verifiability verification on the control strategy. Only when the control strategy meets the verification constraints is the execution phase allowed.

[0055] The edge control terminal generates control policies based on semantic mapping relationships. The control policies record the interface call order, execution parameters, security boundary parameters, abnormal triggering conditions, and verification conditions. For example, in a mine tunnel inspection scenario, the control policies include speed limit, maintaining a safe distance for personnel, detouring around water accumulation along passable areas, and stopping reporting when the path is blocked. In a military boundary shooting scenario, the control policies include moving to a stopping point outside the boundary, maintaining a limited distance, adjusting the gimbal angle, acquiring target images, and stopping due to boundary crossing risks.

[0056] After the control strategy is generated, the edge control terminal performs quality control verification. Task consistency verification checks whether the control strategy covers the target object, target actions, and verification requirements; safety constraint verification checks whether the control strategy avoids restricted boundaries, hazardous areas, and prohibited actions; equipment executability verification checks whether speed, steering, braking, docking, shooting, or reporting parameters are within the current equipment's capabilities; and result verifiability verification checks whether the control strategy includes arrival confirmation, image acquisition confirmation, distance maintenance confirmation, or anomaly reporting confirmation.

[0057] When the control strategy meets the quality control verification, the control strategy is marked as an executable strategy; when the control strategy has speed exceeding the limit, docking point exceeding the boundary, insufficient shooting distance, missing abnormal stopping conditions, or missing verification conditions, the control strategy will not enter the execution stage and a corresponding correction mark will be generated.

[0058] S60. The control strategy that has passed the quality control verification is converted into the underlying control command of the unmanned equipment and issued for execution. Based on the execution feedback, the environmental feature description and prompt word retrieval results are dynamically updated to trigger the reconstruction of control intent and the correction of control strategy, forming a training-free adaptive control closed loop based on fixed model weights.

[0059] The edge control terminal converts the control strategy verified by quality control into underlying control commands and issues them for execution. Simultaneously, it updates the environmental feature description and prompt word search results based on execution feedback, including the following sub-steps: S601. Convert the control strategy that has passed the quality control verification into motion control commands, attitude control commands, task execution commands, abnormal stop commands or rollback control commands that can be recognized by the underlying control interface of the unmanned equipment, and send them to the corresponding unmanned equipment for execution. The edge control terminal generates underlying control commands based on the interface call order, execution parameters, security boundary parameters, and abnormal triggering conditions in the control strategy.

[0060] In the scenario of mine roadway inspection, "progress at low speed, avoid water accumulation, maintain a safe distance for personnel, and stop reporting when the path is blocked" is converted into speed limit movement instructions, path following instructions, obstacle avoidance distance control instructions, and abnormal stop reporting instructions; in the scenario of shooting outside the military industrial boundary, "move to the docking point outside the boundary, adjust the gimbal angle, acquire target image, and stop if there is a risk of crossing the boundary" is converted into fixed-point movement instructions, gimbal attitude adjustment instructions, image acquisition instructions, and boundary crossing stop instructions.

[0061] When a safety boundary or abnormal condition is triggered during execution, the edge control terminal issues an abnormal stop command or rollback control command and records the triggering reason.

[0062] S602. Collect visual feedback, position feedback, attitude feedback, equipment status feedback and task result feedback during the execution of unmanned equipment, and update the environmental feature description and prompt word retrieval results based on the feedback results to trigger control intent reconstruction and control strategy correction, forming a training-free adaptive control closed loop.

[0063] The edge control terminal collects visual feedback, position feedback, posture feedback, device status feedback, and task result feedback, and compares the feedback results with the verification requirements in the structured control intent.

[0064] When the feedback results show that there are new obstacles, people occupying the path, target loss, path blockage, increased risk of crossing boundaries, or abnormal equipment status, the edge control terminal updates the risk labels, action constraints, and equipment status in the environmental feature description and rematches the prompt word template.

[0065] The edge control terminal regenerates the control intent and corrects the control strategy based on the updated prompt word search results; when the feedback results meet the verification requirements, the task completion status is recorded.

[0066] Example 2 like Figure 2 As shown, Embodiment 2 of this application provides a training-free adaptive control system for unmanned equipment based on knowledge enhancement and quality control, comprising: The scene data acquisition module is used to collect visual input, mission commands, and equipment status information of heterogeneous unmanned equipment after entering a new operating environment, and to fuse these data to generate scene data to be adapted. This module includes: The visual acquisition submodule is used to acquire visual images, video streams, and environmental spatial structure information, and to identify target objects, scene boundaries, obstacle distribution, and passable areas, forming a basic semantic representation of the scene.

[0067] The state fusion submodule is used to synchronously collect task instructions, equipment motion status, load status, communication status and equipment execution constraint parameters, and fuse task objectives, equipment execution conditions and scene semantic basic representation to generate scene data to be adapted.

[0068] The environment feature generation module is used to input the scene data to be adapted into a lightweight VLM deployed on the edge with fixed model weights, and generate environment feature descriptions to trigger control logic selection. This module includes: The semantic parsing submodule is used to input the scene data to be adapted into the lightweight VLM. The lightweight VLM performs multimodal semantic parsing on the visual input, task instructions and equipment status information, and keeps the model weights unchanged during the on-site deployment process.

[0069] The feature output submodule is used to extract scene type, target object, obstacle distribution, dangerous area, passable area and equipment status based on the multimodal semantic parsing results, and generate environmental feature descriptions.

[0070] The knowledge-enhanced retrieval module dynamically retrieves matching prompt word templates and environmental constraint rules from the external scene prompt word library and environment knowledge base based on environmental feature descriptions, forming knowledge-enhanced prompt information. This module includes: The template retrieval submodule is used to perform semantic retrieval of the cheat scene prompt word library based on environmental feature descriptions, and match prompt word templates that correspond to the current scene type, target object, task objective and device execution conditions.

[0071] The rule invocation submodule is used to retrieve the operation rules, safety constraints, action boundaries, exception handling rules and result verification requirements corresponding to the current scenario from the environmental knowledge base, and combine them with the matched prompt word template to form knowledge-enhanced prompt information.

[0072] The structured control intent generation module uses knowledge-enhanced prompts as input to a fixed-weight lightweight VLM to generate structured control intents. This module includes: The logic switching submodule is used to input knowledge-enhanced prompts, environmental feature descriptions, and task instructions into a fixed-weight lightweight VLM, and to use prompt word templates to constrain the scene understanding method and control inference path in order to switch the control logic structure under different scenarios.

[0073] The intent output submodule is used to generate structured control intents from a fixed-weight lightweight VLM without model parameter updates or on-site fine-tuning. The structured control intents include target objects, target actions, action paths, safety boundaries, prohibited actions, exception handling rules, and verification requirements.

[0074] The control strategy generation and quality control module is used to establish a mapping relationship between action semantics and equipment control interfaces based on structured control intentions and heterogeneous unmanned equipment capability dictionaries, generate control strategies, and perform quality control verification. This module includes: The action mapping submodule is used to call the heterogeneous unmanned equipment capability dictionary to determine the motion capabilities, perception capabilities, execution capabilities, communication capabilities and underlying control interface types supported by the current unmanned equipment, and to establish a semantic mapping relationship between the target action and the underlying control interface.

[0075] The quality control verification submodule is used to generate control strategies based on semantic mapping relationships, and to perform task consistency verification, security constraint verification, equipment executability verification, and result verifiability verification on the control strategies. It is only allowed to enter the execution phase when the control strategy meets the verification constraints.

[0076] The execution and closed-loop feedback module is used to convert the control strategy that has passed quality control verification into low-level control commands for unmanned equipment and issue them for execution. It also dynamically updates the environmental feature descriptions and prompt word search results based on the execution feedback. This module includes: The instruction execution submodule is used to convert the control strategy that has passed the quality control verification into motion control instructions, attitude control instructions, task execution instructions, abnormal stop instructions or rollback control instructions, and send them to the corresponding unmanned equipment for execution.

[0077] The feedback correction submodule is used to collect visual feedback, position feedback, posture feedback, device status feedback and task result feedback, and update the environmental feature description and prompt word retrieval results based on the feedback results to regenerate the control intent and correct the control strategy, forming a training-free adaptive control closed loop.

[0078] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control, characterized in that, Includes the following steps: S10. Collect visual input, task instructions and equipment status information of heterogeneous unmanned equipment after entering a new working environment, and integrate them to generate scene data to be adapted, which represents the semantic structure of the current scene, task objectives and equipment execution constraints. S20. Input the scene data to be adapted into a lightweight VLM deployed on the edge side with fixed model weights, so that the lightweight VLM is only used for scene semantic parsing and does not participate in model parameter updates or fine-tuning training, and generates an environmental feature description for triggering control logic selection. S30. Based on the description of environmental features, the system dynamically retrieves and matches prompt word templates and environmental constraint rules from the external scene prompt word library and environment knowledge base through a retrieval enhancement mechanism, and uses the prompt word templates as the constraint source for the control logic generation structure to form knowledge-enhanced prompt information. S40. Input the knowledge-enhanced prompt information as the control generation constraint into the fixed-weight lightweight VLM, so that the lightweight VLM can generate structured control intent based on the control structure defined by the prompt word template without retraining. S50. Based on the structured control intent and the heterogeneous unmanned equipment capability dictionary, establish the mapping relationship between action semantics and equipment control interface to generate control strategy. Before execution, perform task consistency constraint verification, safety constraint verification and equipment executability verification on the control strategy. Only when the verification constraints are met is the execution phase allowed. S60. The control strategy that has passed the quality control verification is converted into the underlying control command of the unmanned equipment and issued for execution. Based on the execution feedback, the environmental feature description and prompt word retrieval results are dynamically updated to trigger the reconstruction of control intent and the correction of control strategy, forming a training-free adaptive control closed loop based on fixed model weights.

2. The training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control according to claim 1, characterized in that, The process involves collecting visual input, task commands, and equipment status information from heterogeneous unmanned equipment after it enters a new operational environment, and fusing these data to generate adaptive scenario data that characterizes the semantic structure of the current scene, task objectives, and equipment execution constraints. This includes the following sub-steps: S101. Collect visual images, video streams and environmental spatial structure information after the heterogeneous unmanned equipment enters the new operating environment, and identify target objects, scene boundaries, obstacle distribution and passable areas in the current environment to form a basic semantic representation of the scene. S102. Synchronously collect task instructions, equipment motion status, load status, communication status and equipment execution constraint parameters, and integrate task objectives, equipment execution conditions and scene semantic basic representation to generate scene data to be adapted.

3. The training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control according to claim 1, characterized in that, The scene data to be adapted is input into a lightweight VLM deployed on the edge with fixed model weights. This lightweight VLM is used only for scene semantic parsing and does not participate in model parameter updates or fine-tuning training. This generates an environmental feature description for triggering control logic selection, including the following sub-steps: S201. Input the scene data to be adapted into the lightweight VLM deployed on the edge side. The lightweight VLM performs multimodal semantic parsing on the visual input, task instructions and equipment status information, and keeps the model weights unchanged during the on-site deployment process. S202. Extract scene type, target object, obstacle distribution, dangerous area, passable area and equipment status based on the multimodal semantic parsing results, and generate an environmental feature description to trigger subsequent prompt word retrieval and control logic selection.

4. The training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control according to claim 1, characterized in that, Based on environmental feature descriptions, a retrieval enhancement mechanism dynamically retrieves and matches prompt word templates and environmental constraint rules from the cheat scene prompt word library and environment knowledge base. The prompt word templates are used as the constraint source for the control logic generation structure to form knowledge-enhanced prompt information, including the following sub-steps: S301. Based on the environmental feature description, perform semantic retrieval of the external scene prompt word library and match prompt word templates corresponding to the current scene type, target object, task target and device execution conditions; S302. Retrieve the operation rules, safety constraints, action boundaries, abnormal handling rules and result verification requirements corresponding to the current scenario from the environmental knowledge base, and combine them with the matched prompt word template to form knowledge-enhanced prompt information.

5. The training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control according to claim 1, characterized in that, The knowledge-enhanced prompts are used as control generation constraints input to a fixed-weight lightweight VLM, enabling the lightweight VLM to generate structured control intentions based on the control structure defined by the prompt word template without retraining. This includes the following sub-steps: S401. Input the knowledge-enhanced prompts, environmental feature descriptions, and task instructions into the fixed-weight lightweight VLM. Use prompt word templates to constrain the scene understanding method and control inference path of the lightweight VLM to trigger the switching of control logic structure under different scenarios. S402. A structured control intent is generated by a fixed-weight lightweight VLM without model parameter updates or on-site fine-tuning. The structured control intent includes the target object, target action, action path, safety boundary, prohibited actions, anomaly handling rules, and verification requirements.

6. The training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control according to claim 1, characterized in that, Based on the structured control intent and the heterogeneous unmanned equipment capability dictionary, a mapping relationship between action semantics and equipment control interfaces is established to generate a control strategy. Before execution, the control strategy is subjected to task consistency constraint verification, safety constraint verification, and equipment executability verification, including the following sub-steps: S501. Based on the structured control intent, call the heterogeneous unmanned equipment capability dictionary to determine the motion capability, perception capability, execution capability, communication capability and underlying control interface type supported by the current unmanned equipment, and establish the semantic mapping relationship between the target action and the underlying control interface. S502. Generate a control strategy based on the semantic mapping relationship, and perform task consistency verification, security constraint verification, device executability verification, and result verifiability verification on the control strategy. Only when the control strategy meets the verification constraints is the execution phase allowed.

7. The training-free adaptive control method for unmanned equipment based on knowledge enhancement and quality control according to claim 1, characterized in that, The control strategy, verified by quality control, is converted into low-level control commands for unmanned equipment and executed. Based on the execution feedback, the environmental feature description and prompt word retrieval results are dynamically updated to trigger control intent reconstruction and control strategy correction, forming a training-free adaptive control closed loop based on fixed model weights. This includes the following sub-steps: S601. Convert the control strategy that has passed the quality control verification into motion control commands, attitude control commands, task execution commands, abnormal stop commands or rollback control commands that can be recognized by the underlying control interface of the unmanned equipment, and send them to the corresponding unmanned equipment for execution. S602. Collect visual feedback, position feedback, attitude feedback, equipment status feedback and task result feedback during the execution of unmanned equipment, and update the environmental feature description and prompt word retrieval results based on the feedback results to trigger control intent reconstruction and control strategy correction, forming a training-free adaptive control closed loop.

8. A training-free adaptive control system for unmanned equipment based on knowledge enhancement and quality control, characterized in that, include: The scene data acquisition module is used to collect visual input, task instructions and equipment status information after heterogeneous unmanned equipment enters a new operating environment, and integrate them to generate scene data to be adapted. The environment feature generation module is used to input the scene data to be adapted into a lightweight VLM deployed on the edge with fixed model weights, and generate an environment feature description for triggering the selection of control logic. The knowledge-enhanced retrieval module is used to dynamically retrieve matching prompt word templates and environmental constraint rules from the plug-in scene prompt word library and environment knowledge base based on environmental feature descriptions, and form knowledge-enhanced prompt information; The structured control intent generation module is used to input knowledge-enhanced prompts as control generation constraints into a fixed-weight lightweight VLM to generate structured control intents. The control strategy generation and quality control module is used to generate control strategies by establishing a mapping relationship between action semantics and equipment control interfaces based on structured control intentions and heterogeneous unmanned equipment capability dictionaries, and to perform quality control verification on the control strategies in terms of task consistency, safety constraints, equipment executability and result verifiability. The execution and closed-loop feedback module is used to convert the control strategy that has passed the quality control verification into the underlying control commands of the unmanned equipment and issue them for execution. Based on the execution feedback, it dynamically corrects the environmental feature description, prompt word retrieval results, control intention and control strategy to form a training-free adaptive control closed loop based on fixed model weights.

Images