Robot control method and device based on semantic action integration, and storage medium

Abstract

The application discloses a robot control method and device based on semantic action integration and a storage medium, relates to the technical field of robot control, and comprises the following steps: based on a preset multimodal model, performing semantic analysis and multimodal fusion on environmental information and language instructions to generate a semantic condition; based on a preset control cycle, generating an action path according to the semantic condition and robot state information; and sending the action path to the robot to control the robot to perform corresponding actions based on the action path. Through the event-driven low-frequency semantic path processing and the differentiated operation mechanism generated by the time cycle control high-frequency action path, for non-key semantic changes of a target object, the VLM inference link does not need to be restarted, and only the high-frequency action layer makes local correction to the action path within a given semantic condition framework, so that response time is reduced and the system adaptability in a dynamic environment is improved.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a robot control method, device and storage medium based on semantic-action integration. Background Technology

[0002] To enable robots to perform complex tasks such as grasping, cleaning, navigation, and manipulation in real-world physical environments, a serial link approach using a Visual Language Model (VLM) and controller is typically employed. This approach integrates visual perception, language understanding, and control execution, providing an effective framework for robots to perform complex tasks in real-world physical environments. In this VLM-controller serial link approach, the robot captures information about its surroundings through a camera. The VLM receives the raw information from the camera and uses natural language processing (NLP) technology to process and analyze this information, extracting useful features and instructions. Based on the processed information, the VLM generates corresponding instructions. The controller receives the instructions from the VLM and generates control signals accordingly. These signals are sent to the robot's actuators (such as motors and drives) to drive the robot to perform the corresponding task.

[0003] In the aforementioned serial architecture, the semantic output of the VLM directly determines the controller's action path, with no independent adjustment space between the two. The VLM outputs a unified semantic-action binding result, rather than separable semantic information and independent action path templates. However, in real-world dynamic environments, semantic changes occur frequently, such as target object position shifts, the addition of environmental obstacles, and fine-tuning of task intent. Any of these semantic changes requires the VLM to re-execute the entire inference process—image input, feature recognition, and path planning—and cannot make local corrections based on the original inference results. Every semantic change necessitates restarting the VLM inference chain, increasing overall response time and resulting in poor dynamic adaptability.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a robot control method, device and storage medium based on semantic-action integration, which aims to solve the technical problem of how to reduce the response time of the reasoning process of visual language model and improve the reasoning efficiency.

[0006] To address the aforementioned problems, this application provides a robot control method based on semantic-action integration, which includes: Based on a pre-defined multimodal model, semantic parsing and multimodal fusion are performed on environmental information and language instructions to generate semantic conditions; Based on a preset control cycle, an action path is generated according to the semantic conditions and robot state information; The motion path is sent to the robot, and the robot is controlled to perform the corresponding action based on the motion path.

[0007] In one embodiment, before the step of generating a motion path based on the semantic conditions and robot state information according to a preset control cycle, the robot control method based on semantic-motion integration further includes: When one or more of the following are detected: task execution deviation, environmental change, or robot state abnormality, the preset semantic update triggering conditions are confirmed to be met, and a semantic update signal is generated. The semantic conditions are updated based on the semantic update signal to obtain the updated semantic conditions.

[0008] In one embodiment, the step of confirming that the preset semantic update triggering condition is met when one or more of the following are detected: task execution deviation, environmental change, or robot state abnormality, includes: Based on the current environmental information and the environmental semantic information in the previous semantic cycle, determine whether there are changes in the target object, changes in the environmental structure, and / or changes in the passable area. If so, determine that the environmental changes exist. Based on the robot's execution feedback information, it is determined whether the actual motion result deviates from the expectation, the success rate of motion generation decreases, and / or there is a safety intervention operation in the motion path. If so, it is determined that there is a deviation in the task execution. When a safety constraint is triggered, the actuator state becomes abnormal, and / or the speed or attitude reaches a preset safety boundary threshold, it is determined that the robot state is abnormal.

[0009] In one embodiment, the step of semantically parsing and multimodal fusing environmental information and language instructions based on a preset multimodal model to generate semantic conditions includes: Extract basic visual features and task semantic features from the environmental information, and associate the basic visual features and the task semantic features to obtain visual semantic features; Semantic parsing of natural language instructions yields linguistic semantic features; The semantic conditions are obtained by weighted fusion and vectorization of the linguistic semantic features and the visual semantic features.

[0010] In one embodiment, the step of generating a motion path based on a preset control cycle, according to the semantic conditions and robot state information, includes: When the control cycle is reached, the robot state information and the semantic conditions are combined and encoded to obtain a conditional representation; Based on the condition representation and the preset time window, a short-term action change trend model is performed to obtain a short-term action block; For short-duration action blocks generated in adjacent time windows, smoothing and action block splicing are performed to determine the resulting continuous action sequence as the action path.

[0011] In one embodiment, the step of modeling short-term action change trends based on the condition representation and a preset time window to obtain short-term action blocks includes: The condition representation is matched with a preset trend mapping rule library, and the movement trend corresponding to the condition representation is determined based on the target trend mapping rule obtained from the match. Based on a preset time-series generation model, the motion trend is feature extracted and encoded to obtain a trend feature vector; According to the time series pattern in the time series generation model, the trend feature vector is mapped to a time series parameter sequence; The time sequence parameter sequence is divided into sub-sequences based on the time window, and the sub-sequences are determined as the short-time action blocks.

[0012] In one embodiment, the robot control method based on semantic-action integration further includes: Obtain feedback information from the robot during the execution of the action path, and determine whether there are any abnormal situations based on the feedback information; If the aforementioned abnormal situation exists, the corresponding security intervention operation will be triggered according to the abnormality type of the abnormal situation; The semantic conditions and / or the action path are updated based on the security intervention operation.

[0013] In one embodiment, the step of obtaining feedback information from the robot during the execution of the motion path and determining whether there is an abnormality based on the feedback information includes: The motion deviation is determined based on the expected motion state data and the actual motion state data. If the motion deviation is greater than or equal to a preset deviation threshold, the abnormal situation is confirmed to exist. Obtain the rate of change of motion parameters in adjacent cycles and the variance of motion parameters in a preset cycle. If the rate of change is greater than or equal to a preset rate of change threshold and / or the variance of ... The actuator operating data of the robot is compared with the corresponding normal operating parameter range. If the actuator operating data is not within the normal operating parameter range, the abnormal situation is confirmed. If the distance between a path point and an obstacle point is less than the preset minimum safe distance, then the abnormal situation is considered to exist.

[0014] Furthermore, to achieve the above objectives, this application also proposes a robot control device based on semantic-action integration, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot control method based on semantic-action integration as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the robot control method based on semantic-action integration as described above.

[0016] This application provides a robot control method based on semantic-action integration. It performs semantic parsing and fusion of environmental information and language instructions based on a multimodal model to generate semantic conditions. Then, according to a preset control cycle, it combines the semantic conditions with real-time robot state information to generate fine-grained, real-time adaptable action paths at high frequency. Finally, the action paths are sent to the robot and driven to execute. This method not only achieves stable implementation of high-level semantic constraints, but also ensures flexible response of low-level actions to real-time states and dynamic environments, achieving the overall technical effect of accurate task objectives, continuous and safe execution, and optimized computing power allocation.

[0017] In addition, by using event-driven low-frequency semantic path processing and time-cycle controlled high-frequency action path generation, the VLM inference link does not need to be restarted for non-critical semantic changes such as minor target object offsets, local changes in obstacles, and actuator load fluctuations. Instead, the high-frequency action layer makes local corrections to the action path within the given semantic condition framework, which reduces response time and improves the system's adaptability in dynamic environments. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1This is a first flowchart illustrating the robot control method based on semantic-action integration provided in this application. Figure 2 A second flowchart illustrating the robot control method based on semantic-action integration provided in this application; Figure 3 This is a schematic diagram of the hardware operating environment involved in the robot control method based on semantic-action integration in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] To achieve the above objectives, this application proposes a robot control method based on semantic-action integration. The method includes: performing semantic parsing and multimodal fusion on environmental information and language instructions based on a preset multimodal model to generate semantic conditions; generating action paths based on the semantic conditions and robot state information according to a preset control cycle; and sending the action paths to the robot to control the robot to perform corresponding actions based on the action paths.

[0025] To enable robots to perform complex tasks such as grasping, cleaning, navigation, and manipulation in real-world physical environments, a serial link approach using a Visual Language Model (VLM) and a controller is typically employed. This approach integrates visual perception, language understanding, and control execution, providing an effective framework for robots to perform complex tasks in real-world environments. In this VLM and controller serial link approach, the robot captures information about its surroundings through a camera. The VLM receives the raw information from the camera and uses natural language processing (NLP) technology to process and analyze this information, extracting useful features and instructions. Based on the processed information, the VLM generates corresponding instructions. The controller receives these instructions from the VLM and generates control signals accordingly. These signals are then sent to the robot's actuators (such as motors and drives) to drive the robot to perform the corresponding task.

[0026] In the aforementioned serial architecture, the semantic output of the VLM directly determines the controller's action path, with no independent adjustment space between the two. The VLM outputs a unified semantic-action binding result, rather than separable semantic information and independent action path templates. However, in real-world dynamic environments, semantic changes occur frequently, such as target object position shifts, the addition of environmental obstacles, and fine-tuning of task intent. Any of these semantic changes requires the VLM to re-execute the entire inference process—image input, feature recognition, and path planning—and cannot make local corrections based on the original inference results. Every semantic change necessitates restarting the VLM inference chain, increasing overall response time and resulting in poor dynamic adaptability.

[0027] This application provides a robot control method based on semantic-action integration. It performs semantic parsing and fusion of environmental information and language instructions based on a multimodal model to generate semantic conditions. Then, according to a preset control cycle, it combines the semantic conditions with real-time robot state information to generate fine-grained, real-time adaptable action paths at high frequency. Finally, the action paths are sent to the robot and driven to execute. This method not only achieves stable implementation of high-level semantic constraints, but also ensures flexible response of low-level actions to real-time states and dynamic environments, achieving the overall technical effect of accurate task objectives, continuous and safe execution, and optimized computing power allocation.

[0028] In addition, by using event-driven low-frequency semantic path processing and time-cycle controlled high-frequency action path generation, the VLM inference link does not need to be restarted for non-critical semantic changes such as minor target object offsets, local changes in obstacles, and actuator load fluctuations. Instead, the high-frequency action layer makes local corrections to the action path within the given semantic condition framework, which reduces response time and improves the system's adaptability in dynamic environments.

[0029] It should be noted that the executing entity in this embodiment can be a computing service device with network communication and program execution functions, such as a robot, unmanned vehicle, tablet computer, personal computer, mobile phone, etc., or an electronic device or apparatus capable of realizing the above functions. The following description uses a robot control device based on semantic-action integration as an example to illustrate this embodiment and the subsequent embodiments.

[0030] Based on this, embodiments of this application provide a robot control method based on semantic-action integration, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the robot control method based on semantic-action integration of this application.

[0031] In this embodiment, the robot control method based on semantic-action integration includes steps S10-S30: Step S10: Based on the preset multimodal model, perform semantic parsing and multimodal fusion on environmental information and language instructions to generate semantic conditions.

[0032] It should be noted that environmental information is used to reflect the spatial structure of the robot's environment and the distribution of target objects, while language instructions are used to describe the task objectives and execution constraints that the robot needs to complete.

[0033] In this embodiment, the robot acquires image / point cloud data of the physical environment through visual sensors such as cameras and depth cameras. The images and / or point cloud data are then input as environmental information into a pre-trained Visual Language Model (VLM) to achieve semantic understanding. Optionally, images and depth point cloud data acquired at the same time are associated using timestamp matching. The acquired images undergo operations such as denoising, contrast enhancement, and brightness adjustment. For example, Gaussian filtering is used to remove noise from the image, and histogram equalization is used to enhance image contrast. The depth point cloud data is filtered to remove noise points and outliers, and point cloud downsampling is performed to reduce data volume and improve processing efficiency. For example, statistical filters are used to remove outliers, and voxel grid filters are used for downsampling. The pre-processed image data and depth point cloud data are fused, combining the color information in the image with the three-dimensional coordinate information in the point cloud to obtain the fused environmental information.

[0034] Based on the fused environmental information, the system distinguishes between task objects, obstacles, and interactive areas. For example, task objects could be a cup to be grasped, a table to be cleaned, etc., while obstacles could be cables on the ground, tables and chairs in the corner, etc. The detection model is trained using a pre-labeled environmental feature dataset containing information such as task objects, obstacles, and interactive areas. The detection model can be a deep learning-based object detection model, such as Faster R-CNN or YOLO. After training, the extracted image and point cloud features are input into the trained model, and the model outputs information such as the category, location, and bounding box of each object in the environment. Based on the recognition results, semantic annotation is performed on objects in the environment, assigning a corresponding semantic label to each object to obtain a structured semantic space association dataset. For example: "Target Object": [{"Name": "Red Water Cup", "Semantic Label": "Grasp Target", "Spatial Coordinates": (X1, Y1, Z1), "Size": (W1, H1, D1)}], "Obstacle Object": [{"Name": "Mobile Phone", "Semantic Label": "Obstacle Avoidance Target", "Spatial Coordinates": (X2, Y2, Z2), "Area Range": "Right 1 / 3 of Desktop"}], "Interactive Area": ​​[{"Name": "Desktop", "Semantic Label": "Operating Platform", "Spatial Area": ​​(...] Xmin Y min Zmin )-(X max Y max Z max )}).

[0035] Based on pre-trained natural language processing models such as BERT and GPT, key elements such as target objects, location information, and action types in natural language instructions are analyzed to obtain a structured task target set. This task target set is then matched with a semantic space-related dataset to generate an action logic sequence. This sequence represents semantic-level action steps and does not include quantized parameters. Finally, the action logic is transformed into a machine-understandable semantic action instruction set, i.e., semantic conditions.

[0036] In one feasible implementation, step S10 includes steps S11 to S13: Step S11: Extract the basic visual features and task semantic features from the environmental information, and associate the basic visual features and the task semantic features to obtain visual semantic features.

[0037] In this embodiment, structured feature extraction is performed on the collected environmental information. A pre-defined visual feature extraction network / algorithm is used to extract low-dimensional, quantifiable basic features from the raw environmental data, including: Geometric features such as object outline, edges, corners, depth distribution, spatial coordinates, and size ratio; texture features such as color histogram, texture gradient, and pixel grayscale distribution; and motion features such as target trajectory, speed, and direction are all considered. All basic visual features are converted into numerical vectors of a unified dimension, such as normalization to the 0-1 range, and redundant features such as irrelevant background pixels are removed to generate a basic visual feature vector. Based on a preset task scenario (such as grasping, navigation, or cleaning), semantic features strongly related to the task are extracted from environmental information. Through object detection algorithms, core objects related to the task in the environment are identified, and the object category, attributes, and status are output. The spatial and interaction relationships between core objects are analyzed. The identified object categories, attributes, and spatial / interaction relationships are converted into discrete numerical / label vectors according to preset encoding rules, such as encoding a water cup as [1,0,0], a desktop as [0,1,0], and above as [0,0,1], generating a task semantic feature vector. The core objects in the task semantic features are bound to the corresponding geometric features in the basic visual features to establish a one-to-one correspondence between semantic objects and visual features; the task-related semantic rules are mapped to the constraint thresholds of the basic visual features; the associated basic visual feature vectors are concatenated with the task semantic feature vectors, and the dimensions are unified through linear transformation to generate visual semantic feature vectors.

[0038] Step S12: Semantically analyze the natural language instructions to obtain language semantic features.

[0039] In this implementation, natural language instructions are preprocessed to remove redundant characters, correct typos, and standardize the sentence structure. Instructions are then broken down into their smallest semantic units, each labeled with its part of speech, and core task words, target object words, attribute words, and constraint words are identified. Task type, target object, object attributes, and task constraints (limitations and spatial / action relationships in the instructions) are extracted from the segmentation results. Using built-in standardized semantic encoding rules, the extracted semantic elements are transformed into computable numerical vectors. These encoded vectors are then concatenated in a fixed order to form a language semantic feature vector.

[0040] Step S13: The linguistic semantic features and the visual semantic features are weighted, fused, and vectorized to obtain the semantic conditions.

[0041] In this embodiment, the two types of feature vectors are weighted and summed according to a preset weighting rule. The weights are dynamically adjusted based on the task scenario. For example, in a vision-dominated task, the weight of visual semantic features is 0.7, and the weight of linguistic semantic features is 0.3; in a linguistic-dominated task, the weight of linguistic semantic features is 0.7, and the weight of visual semantic features is 0.3; in a general scenario, the weight of both types of features is 0.5. For the two dimensionally aligned vectors, the visual feature value × visual weight + linguistic feature value × linguistic weight is applied dimension by dimension to generate the initial fused vector. Through a preset feature optimization algorithm, such as an attention mechanism or a multilayer perceptron, a nonlinear transformation is performed on the initial fused vector to strengthen task-related features and weaken irrelevant features. The optimized fused vector is then combined with a preset semantic labeling system to transform it into structured semantic conditions.

[0042] Step S20: Based on a preset control cycle, generate an action path according to the semantic conditions and robot state information.

[0043] In this embodiment, the robot state information includes kinematic state information, actuator state information, historical state information, and safety-related state information. Kinematic state information is used to characterize the robot's current motion state, including its current position, posture, speed, and their changing trends. Actuator state information is used to characterize the operation of the robot's actuators, including actuator output status, load status, or operational anomaly indication information. Historical state information is used to characterize the robot's historical motion state changes within a preset time window. Safety-related state information is used to characterize whether the robot is currently in a safe operating state, including speed limit status, anomaly flags, or safety constraint trigger information.

[0044] When the preset control cycle is reached, based on the semantic conditions output by VLM and the robot's state information, motion parameterization calculation is performed. For example, moving above the water cup is converted into the rotation angle, movement speed, and acceleration limit of each joint.

[0045] The structured semantic task objective is broken down into multi-stage sub-objectives, each corresponding to specific kinematic requirements. For example, grasping a water cup is broken down into: moving to a safe height above the cup → adjusting the end effector's posture to align with the cup opening → descending to the grasping position → clamping the actuator → raising to a safe height. A quantified benchmark is assigned to each sub-objective, including target position, target posture, target velocity / acceleration, and actuator motion requirements. Semantic constraints are transformed into computer-recognizable decision rules. Safety constraints are transformed into path verification thresholds, such as obstacle avoidance mapping to a distance ≥ preset obstacle avoidance distance; performance constraints are transformed into parameter upper limits, and actuator constraints are transformed into output limits, such as light load mapping to actuator torque ≤ 80% of rated value. A constraint rule table is formed, containing rule type, decision conditions, and post-trigger processing logic.

[0046] The robot's state information is compared with the quantization benchmark of the sub-objective. For kinematic state information, it is determined whether the current position / attitude is within the reachable range of the sub-objective and whether the velocity / acceleration can reach the target value. For actuator state information, it is determined whether the current load of the actuator meets the action requirements of the sub-objective. For safety state information, it is determined whether the current safety state allows the execution of the sub-objective. If a sub-objective is not feasible, the parameters of the sub-objective are adjusted (such as reducing the speed or adjusting the attitude) or a transitional sub-objective is added (such as retreating to a safe position before execution). Secondly, the execution strategy of the sub-objective is modified by combining the historical state within the preset time window. If the historical data shows that the speed fluctuation of a certain joint is greater than the preset speed fluctuation threshold, the upper limit of the acceleration of that joint is reduced and a smooth transition step is added. If the historical actuator overload is frequent, the actuator load requirement of the corresponding sub-objective is reduced and the execution time is extended. This ensures that the path generation adapts to the actual operating characteristics of the robot and avoids repeated triggering of anomalies.

[0047] Following the order of sub-objectives, critical path points are generated between the current state and the endpoints of each sub-objective. Each path point contains core parameters such as position, attitude, velocity, and actuator commands, and conforms to the constraint rule table, forming a coarse path point sequence. Continuous interpolation is performed on adjacent coarse path points to generate fine motion parameters corresponding to each control cycle. Pre-defined interpolation rules, such as linear interpolation and polynomial interpolation, are used to ensure continuous changes in position, attitude, and velocity without abrupt changes. A timestamp is assigned to each interpolation point, with the timestamp interval equal to the control cycle, forming a fine motion parameter sequence with a time axis. For example, with a control cycle of 20ms, a 1-second path corresponds to 50 interpolation points, and each point corresponds to a 20ms time window motion command. Each fine interpolation point is traversed, and item-by-item verification is performed in conjunction with the robot's real-time state information. If a point fails verification, the parameters of that point and subsequent points are immediately adjusted, such as reducing speed, adjusting attitude, or increasing obstacle avoidance offset. The adjusted parameter sequence is then subjected to global smoothing to eliminate local parameter abrupt changes. The final, optimized sequence of fine motion parameters is transformed into a control instruction format that the robot actuator can directly recognize. Kinematic parameters are transformed into joint angle / endpoint pose instructions; velocity / acceleration parameters are transformed into motor speed / torque instructions; actuator state requirements are transformed into actuator switching / load adjustment instructions; and safety states are transformed into safety flags. This yields a motion path ordered by timestamps and perfectly matched to the control cycle, containing timestamps, quantized control parameters, and safety verification results, which can be directly executed sequentially by the robot controller.

[0048] Step S30: Send the motion path to the robot and control the robot to perform the corresponding action based on the motion path.

[0049] In this embodiment, the generated motion path is converted into standardized instructions recognizable by the robot controller. Instructions are split according to control cycles, with each instruction frame corresponding to one control cycle and containing a unique timestamp, actuator control parameters, motion type flag, and security check bit. Instruction frames are encapsulated according to protocols supported by the robot controller, such as ROSTopic / Action, Modbus, and custom TCP / UDP protocols, and communication parameters are set. A pre-caching and frame-by-frame delivery strategy is employed to avoid instruction backlog or loss. Motion paths are split according to a preset cache window to generate instruction batches. The first batch of instructions is sent to the robot controller's local cache; subsequent batches are only sent after successful reception confirmation.

[0050] After receiving a command frame, the robot controller checks whether the length and field positions of the command frame conform to the preset format, discarding frames with incorrect formats. The verified command frames are sorted by timestamp and stored in the controller's execution buffer to ensure that the execution order is consistent with the motion path. The robot controller parses the core parameters in the command frame and maps them to specific actuators, converting position / attitude / velocity commands into target values ​​for each joint actuator, such as the inverse kinematics of joint angles for end-effector poses and the target value of motor speed for velocity commands. Action flags are mapped to the control signals of the corresponding actuators, such as the solenoid valve on / off signal for gripper opening / closing commands and the motor enable signal for pause commands. Safety check bits in the command frame are parsed, and corresponding safety constraints are loaded, such as speed limit triggering reducing the maximum motor speed and emergency stop triggering cutting off actuator power. The robot controller executes the commands synchronously based on a preset control cycle, sending the parsed actuator target values ​​to the driver to drive the actuators to perform actions, and recording the actual execution parameters of each command frame.

[0051] In this embodiment, semantic parsing and fusion of environmental information and language commands are performed based on a multimodal model to generate semantic conditions. Then, according to a preset control cycle, fine-grained, real-time adaptable action paths are generated at high frequency by combining semantic conditions and real-time robot state information. Finally, the action paths are sent to the robot to drive its execution. This achieves stable implementation of high-level semantic constraints and ensures flexible response of low-level actions to real-time states and dynamic environments, achieving the overall technical effect of accurate task objectives, continuous and safe execution, and optimized computing power allocation. Through the differentiated operation mechanism of event-driven low-frequency semantic path processing and time-cycle controlled high-frequency action path generation, non-critical semantic changes such as minor target object offsets, local changes in obstacles, and actuator load fluctuations do not require restarting the VLM inference link. Instead, the high-frequency action layer only makes local corrections to the action path within the established semantic condition framework, reducing response time and improving system adaptability in dynamic environments.

[0052] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S20 includes steps S21 to S23: Step S21: When the control cycle is reached, the robot state information and the semantic conditions are combined and encoded to obtain a condition representation.

[0053] In this embodiment, under a control cycle higher than the semantic path update frequency, continuous and controllable action outputs are generated based on semantic conditions and the robot's own state, enabling the robot to complete stable motion under semantic constraints.

[0054] Data acquisition is triggered in each motion control cycle to read robot state information. This information is then categorized and processed. For numerical states, position and velocity are normalized to the [-1,1] interval to eliminate dimensional differences. For discrete states, anomaly flags and actuator modes are converted into numerical vectors through one-hot encoding. This yields standardized state feature vectors with uniform dimensions and scale. These standardized feature vectors are then concatenated with effective semantic condition vectors along fixed dimensions to form a joint conditional input containing semantic constraints and real-time state. Using preset encoding rules, such as fully connected layers and linear transformation matrices, feature fusion is performed on the joint conditional input to generate a conditional representation.

[0055] Step S22: Based on the condition representation and the preset time window, perform short-term action change trend modeling to obtain short-term action blocks.

[0056] In this embodiment, based on the internal representation of the short-term action change, a short-term action block within the corresponding time window is generated, and the action output is divided into multiple consecutive short-term action blocks according to a preset time length. Each short-time action block describes the robot's continuous motion behavior within a corresponding time window; short-time action blocks are represented as a continuous sequence of velocity, displacement changes, or actuator control quantities. By adopting the short-time action block format, the motion generation process has a clear temporal structure, facilitating subsequent smoothing and safety control.

[0057] The conditional representation is parsed to extract the motion target trend of the current control cycle, including: target displacement vector, acceleration / deceleration magnitude, and attitude. The trend is verified to ensure it meets constraints, such as the rate of change of velocity not exceeding the maximum acceleration and attitude adjustment not exceeding joint limits; if limits are exceeded, the trend magnitude is automatically reduced. A preset mathematical model is selected, such as linear interpolation, cubic polynomial, or exponential smoothing, to describe continuous motion changes within a preset time window. For example, if the target is to increase velocity from 0.1 m / s to 0.2 m / s within 200 ms, a linear model represents this as an acceleration of 0.01 m / s every 20 ms. The model constrains the motion change pattern, avoiding abrupt parameter changes in a single step. The overall motion output is divided into continuous short-time motion blocks according to a preset time length. The number of control steps in each block = window length / control cycle. Each block is assigned a unique ID and time interval. Based on the internally represented mathematical model, the motion parameters for each control cycle within the window are calculated step-by-step. The trend model is discretized into control quantities recognizable by the actuator. Each motion block is encapsulated as a control step sequence, including the timestamp, velocity, displacement, actuator control quantity, and constraint threshold for each step.

[0058] In one feasible implementation, step S22 includes: matching the conditional representation with a preset trend mapping rule library, determining the motion trend corresponding to the conditional representation based on the target trend mapping rule obtained from the matching; extracting and encoding features of the motion trend based on a preset time series generation model to obtain a trend feature vector; mapping the trend feature vector to a time series parameter sequence according to the time series rules in the time series generation model; dividing the time series parameter sequence into sub-sequences based on the time window, and determining the sub-sequences as the short-term action block.

[0059] In this implementation, a standardized trend mapping rule library is pre-constructed as the basis for matching condition representations and motion trends. The rule index dimension includes core features covering the condition representation, such as task type, target position / pose, velocity constraints, and robot current state (e.g., load / safety flags). The rule output dimension corresponds to quantifiable motion trends, such as direction vector, velocity change rate, attitude adjustment rate, and acceleration upper limit. Feature values ​​corresponding to the rule index dimension are extracted from the condition representation, such as task type = grasping, target position = [x,y,z], and robot load = 0.6. If the extracted features are completely consistent with the index dimension of a rule, the rule is directly matched. If there is no completely matching rule, similarity calculation is used, such as cosine similarity matching the closest rule, or rule interpolation is used to generate an adaptive trend. The motion trend obtained by matching is verified to ensure it conforms to the robot hardware constraints. If it exceeds the limits, the trend parameters are corrected according to the constraint threshold. The matched and corrected rule output dimension parameters are integrated into a structured motion trend set.

[0060] Based on pre-trained temporal generation models, such as Recurrent Neural Networks (RNNs), Temporal Convolutional Networks (TCNs), and linear temporal models, temporal features of motion trends are extracted, such as the changing patterns and temporal correlations of trends. Unstructured trend parameters are transformed into fixed-dimensional numerical vectors. Fixed-dimensional trend feature vectors are obtained based on the quantization parameters of the motion trend set. Parameters such as direction vector, velocity change rate, and attitude adjustment rate are concatenated in a fixed order to transform into a model input vector. The temporal generation model extracts temporal features from the input vector to capture the changing patterns of trends over time. The extracted temporal features are encoded into fixed-dimensional trend feature vectors, with each dimension of the vector corresponding to a temporal feature.

[0061] The process reads the built-in temporal patterns of the time series generation model, such as parameter change rules within the time window, control cycle step size, and parameter interpolation methods. Core patterns include: the number of control steps within the time window, parameter interpolation rules, and inter-step parameter change constraints. Based on these temporal patterns, the trend feature vector is mapped inversely to a continuous temporal parameter sequence within the time window. Through the decoding layer of the time series model, the trend feature vector is transformed into initial parameters for each step within the window. Parameters for all control steps are completed according to the interpolation rules, ensuring the parameter sequence is continuous without gaps. Each step's parameters are checked to ensure they conform to the motion trend constraints; if they exceed the limits, the parameters for the corresponding step are corrected. Finally, a temporal parameter sequence matching the time window is obtained.

[0062] The system determines the preset short-term time window length, the number of control steps within the window, and the window timestamp range. The complete timing parameter sequence is divided into several subsequences according to the window length, with each subsequence corresponding to a time window. Each step parameter in the subsequence is converted into a control quantity recognizable by the robot actuator. A short-term action block is generated for each subsequence, including: action block ID, time window range, number of control steps, subsequence (timestamp of each step, actuator control quantity, parameter constraints), and constraint threshold. A preset rule base is used to match conditional representations with motion trends, avoiding the randomness of trend decisions and adapting to different task scenarios. Hardware constraints are directly verified during the matching process to avoid situations where the trend is feasible but the hardware is not executable.

[0063] Step S23: Smooth the short-time action blocks generated by adjacent time windows and splice the action blocks to determine the resulting continuous action sequence as the action path.

[0064] In this embodiment, the parameters of the last step of the previous action block and the first step of the current action block are extracted, the parameter difference is calculated, and compared with a preset mutation threshold to determine whether there is a mutation trend. If a mutation is detected, the starting parameters of the current block are corrected using exponential smoothing / linear transition; the rate of change of parameters in subsequent control steps within the current block is adjusted to ensure that the target state within the window can still be achieved; the adjusted parameters are verified to meet all constraints to avoid exceeding limits due to smoothing. All smoothed action blocks are concatenated in timestamp order, and duplicate control steps are deleted to form a continuous action sequence without breaks.

[0065] In this embodiment, short-time action blocks are divided according to fixed time windows. The action parameters of each block can be independently verified and adjusted, which facilitates subsequent safety control. For example, if a block triggers obstacle avoidance, only that block can be adjusted. Through internal representation modeling and inter-block smoothing, the robot's actions are continuous in terms of speed, acceleration, and trajectory, without sudden acceleration or change of direction, thus avoiding mechanical vibration or impact.

[0066] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S30, steps A10 to A20 may be included: Step A10: When one or more of the following are detected: task execution deviation, environmental change, or robot state abnormality, the preset semantic update triggering conditions are confirmed to be met, and a semantic update signal is generated.

[0067] In this embodiment, the task semantics are parsed and structured at a relatively low update frequency. The low-frequency semantic path is triggered by an event and remains unchanged when the task objective or environmental semantics do not change significantly, and is not updated with each action control cycle.

[0068] The robot's actual execution state (current position, velocity, and posture) is compared with the target state in the original semantic conditions. The specific values ​​of position deviation, velocity deviation, and posture deviation are calculated, and it is recorded whether the deviation continuously exceeds a preset threshold. The visual semantic features of the current environment (target object position, attributes, traversable area, obstacle distribution) are compared in real-time with the initial environmental features to detect whether the target object has shifted, is missing, or its attributes have changed; whether new obstacles have been added; whether the traversable area has shrunk; and whether environmental data such as lighting and depth have undergone abrupt changes. Various abnormal markers in the robot's status information are analyzed to identify whether the actuator is overloaded, stalled, exceeds temperature limits, or experiences communication failure; whether safety conditions have triggered speed limits, emergency stop warnings, or insufficient human-robot safety distance; and whether kinematic issues such as joint limit triggering, excessive velocity and acceleration, or trajectory jitter have occurred, and the abnormality level is marked.

[0069] The triggering condition is met when any one or more of the following are detected: persistent deviation from task execution limits, critical environmental changes affecting task execution, or moderate to severe robot anomalies. Optionally, to avoid false triggers caused by momentary data fluctuations, deviations, environmental changes, and abnormal signals are filtered, and a valid trigger is only considered when the anomaly or deviation persists for a preset number of control cycles. Different triggering thresholds can be configured for different task types; for example, the positional deviation threshold for a precise grasping task can be set to 0.2 cm, and the positional deviation threshold for a navigation task can be set to 0.5 cm, ensuring that the triggering determination aligns with the task characteristics.

[0070] Once the triggering conditions are met, a semantic update signal is generated. The signal includes the signal type, triggering reason, urgency level, and timestamp. The signal type specifies the triggering source, including deviation triggering, environmental change triggering, robot anomaly triggering, or a combination of multiple types. The triggering reason describes the core issue of the trigger. The urgency level is divided into mild, moderate, and severe according to the degree of impact. The control cycle number corresponding to the triggering time is recorded to ensure that the semantic update and the action execution sequence are strictly aligned.

[0071] In one feasible implementation, step A10 includes steps A11 to A13: Step A11: Based on the current environmental information and the environmental semantic information in the previous semantic cycle, determine whether there are changes in the target object, changes in the environmental structure, and / or changes in the passable area. If so, determine that the environmental changes exist.

[0072] In this embodiment, current environmental information is collected according to semantic cycles, and visual semantic features of the environment are extracted, including target object features, environmental structure features, and passable area features; environmental semantic information from the previous semantic cycle is also obtained. Both types of features are uniformly converted into numerical vectors to eliminate dimensional differences. The difference between the current and historical target object features is calculated. If the Euclidean distance of the target object's center coordinates is greater than a preset threshold, it is determined to be a positional shift; if the cosine similarity of the target object's color, shape, and state encoding vectors is less than a preset value, it is determined to be an attribute change; if a target object existed historically but is not currently identified, it is determined to be missing; if any one of the changes in position, attribute, or existence is satisfied, a change in the target object is determined. The spatial coordinates of the current and historical environmental structure features are compared. If the vertex coordinate deviation of static structures such as walls / desktops is greater than a preset threshold, or if static structures are added / disappear, it is determined to be an environmental structure change. The overlap rate of the current and historical passable areas is calculated. If the overlap rate is less than a preset threshold, or if the change in the passable area area is greater than a preset proportion, or if a restricted area is added, it is determined to be a change in the passable area. If any one or more of the following are detected—namely, changes to the target object, changes to the environmental structure, and changes to the passable area—it is determined that there is an environmental change, and the type of change and the specific difference are recorded.

[0073] Step A12: Based on the robot's execution feedback information, determine whether the actual motion result deviates from the expectation, the motion generation success rate decreases, and / or, there is a safety intervention operation in the motion path. If so, it is determined that there is a deviation in the task execution.

[0074] In this embodiment, the robot's execution feedback information is collected according to the control cycle, including actual motion results, action generation success rate and safety intervention records. The actual motion results include the robot's actual position, speed, posture and actuator action completion degree; the action generation success rate includes the number of failed action command generation within a preset time window and the generation time; the safety intervention records include whether safety interventions were triggered in the action path.

[0075] By comparing the actual motion results with the expected results in the semantic conditions, if the Euclidean distance between the actual position and the expected position is greater than the preset threshold, a positional deviation is confirmed; if the difference between the actual speed and the expected speed is greater than the preset threshold, a speed deviation is confirmed; if the difference between the actual attitude angle and the expected attitude angle is greater than the preset threshold, an attitude deviation is confirmed; for actuator completion, if the gripper clamping force / motor torque does not reach the preset ratio of the expected value, and if any deviation lasts for ≥2 control cycles, it is determined that the actual motion result deviates from the expectation.

[0076] To determine if the success rate of action generation has decreased, the success rate of action generation is compared with the success rate of action generation in the current time window and the historical normal window. If the success rate decreases by more than a preset ratio, or the time taken for a single generation exceeds a preset threshold, it is determined that the success rate of action generation has decreased.

[0077] For the determination of safety intervention operations in the action path, if the execution feedback includes safety intervention records such as collision warning triggering path adjustment, obstacle avoidance causing path detour, and manual emergency stop, and the number of interventions is greater than the preset value, it is determined that there is a safety intervention operation.

[0078] When any one or more of the following are detected: actual motion results deviating from expectations, decreased motion generation success rate, or motion path safety intervention, a task execution deviation is determined, and the deviation type and specific value are recorded.

[0079] Step A13: When a safety constraint is triggered, the actuator state is abnormal, and / or the speed or attitude reaches a preset safety boundary threshold, it is determined that the robot state is abnormal.

[0080] In this embodiment, the safety constraint trigger status, actuator status information, and velocity and attitude boundary data are collected in real time. The safety constraint trigger status indicates whether the preset safety constraint has been triggered, such as insufficient human-machine safety distance, collision warning, and low voltage. The actuator status information indicates the output status, load rate, temperature, and abnormal flags of each actuator (motor / servo motor / gripper). The velocity and attitude boundary data indicates the robot's current linear velocity / angular velocity and Euler angle / quaternion attitude value.

[0081] The system checks each condition according to the rules to determine if any abnormal status exists. If the safety constraint status is triggered, it is determined to be a safety constraint triggering abnormality. For actuator status abnormality determination, if the load rate is greater than the preset threshold or the temperature is greater than the preset threshold, it is determined to be an overload or overheating abnormality. When the abnormality flag is overload, stall, or communication failure, it is determined to be an actuator functional abnormality. When the deviation between the output status and the command value is greater than the preset threshold, it is determined to be an output abnormality; as long as any condition is met, it is determined to be an actuator status abnormality. The current speed and attitude are compared with the preset safety boundary threshold. If the linear velocity or angular velocity is greater than or equal to the safety boundary threshold, or the attitude angle is greater than or equal to the safety boundary threshold, it is determined to be a safety boundary threshold reaching abnormality. If any one or more of the following are detected: safety constraint triggering, actuator status abnormality, or speed and attitude reaching the safety boundary threshold, it is determined that there is a robot status abnormality, and the abnormality type and specific value are recorded.

[0082] Step A20: Update the semantic conditions based on the semantic update signal to obtain the updated semantic conditions.

[0083] In this embodiment, semantic update signals are parsed to extract core information such as signal type, triggering reason, and urgency level. The update dimension and priority of semantic conditions are determined, and update directions are matched according to signal type. Deviation triggers update the target parameters in the corresponding semantic conditions; environmental changes trigger update the corresponding scenario constraints; and robot anomalies trigger update the corresponding execution constraints. Update priority is determined according to urgency level: severe triggers are handled first, requiring the current action to be paused and the emergency target switched; moderate triggers are handled second, adjusting the task target or core constraints; and mild triggers are handled last, only fine-tuning the execution parameters without changing the core task target.

[0084] The system invokes a pre-defined semantic update rule base, which is structured by trigger type, update dimension, and update method. For example, to address excessive position deviation, the matching rule corrects the target position parameters, adjusting the original target coordinates based on the actual position and deviation value, according to the direction and magnitude of the deviation. For newly added obstacles in the environment, the matching rule updates path constraints and obstacle avoidance zones, adding the obstacle coordinates to the prohibited area list and adjusting the target path point to avoid the obstacle. For actuator overload, the matching rule reduces speed and acceleration constraints by a preset ratio, such as reducing the speed limit by 20% and the acceleration limit by 30%, to lower the actuator load. For sensor failure, the matching rule strengthens safety constraints, triggering a speed-limiting mode and expanding the safety distance threshold to improve execution safety.

[0085] Based on the matched update rules, the original semantic conditions are specifically modified to generate updated semantic conditions. Optionally, the update method is divided into parameter update and target update. Parameter update is suitable for light and moderate triggers. It extracts quantifiable values ​​such as target parameters and constraint parameters from the original semantic conditions and directly modifies them according to the update rules, keeping the core task type unchanged. For example, the original speed limit of 0.2m / s is lowered to 0.16m / s, the original target position [x1, y1, z1] is modified to [x1-0.008, y1, z1], or obstacle coordinates are added to the prohibited area field. Target update is suitable for heavy triggers. If the environment changes drastically or the robot experiences severe abnormalities, the core task target of the semantic conditions is adjusted. For example, grabbing the red water cup is changed to retreating to a safe position, and navigation to the destination is changed to pausing and waiting for manual intervention, while adding emergency constraints. The updated semantic conditions are checked for completeness and rationality to ensure that the update is effective and conflict-free. Once the check passes, the updated semantic conditions take effect immediately in the next control cycle. At the same time, the computer archives the update log, recording the parameter comparison before and after the update, the triggering reason, and the update time.

[0086] In this embodiment, the semantic understanding and action planning, which were originally completed in one step by the VLM, are decoupled into independent semantic and action layers. The semantic layer only performs the parsing and fusion of environmental information and language instructions through a multimodal model when triggered by events such as task initialization, critical changes in the environment, or fine-tuning of task intent. This generates stable and reusable semantic conditions, providing independent adjustment space for semantics and actions. On the other hand, the action layer operates at high frequency with a fixed control cycle. Based on a stable semantic condition framework, it independently generates and corrects fine-grained action paths in combination with the robot's real-time state information, without relying on the VLM for re-inference. When non-critical semantic changes occur in the real dynamic environment, such as slight shifts in the target object or local changes in obstacles, there is no need to trigger the entire VLM inference process. The high-frequency action layer only makes local corrections to the action path under the constraints of semantic conditions. Only when the change reaches the semantic update threshold is the low-frequency semantic layer triggered to update the semantic conditions, and then the action layer continues to generate adaptive actions at high frequency based on the new semantic conditions. This avoids restarting the VLM inference chain for every semantic change, reducing overall response time. At the same time, it allows the semantic layer to focus on the stable expression of high-level task intent and environmental constraints, while the action layer focuses on the real-time adaptation and dynamic adjustment of the underlying execution, thus effectively solving the problem of poor dynamic adaptability of the original architecture.

[0087] Based on the first embodiment of this application, in the fourth embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the above-described robot control method based on semantic-action integration may include steps B10-B30: Step B10: Obtain feedback information from the robot during the execution of the motion path, and determine whether there are any abnormalities based on the feedback information.

[0088] In this embodiment, during the execution of the action, robot execution feedback information is collected in real time. This feedback information includes: actual movement position or posture changes, actual movement speed or actuator operating status, and abnormal status information generated during execution. Based on the collected execution feedback information, the robot's current execution status is monitored to determine whether the actual execution status deviates from the expected action sequence; whether the actuator operating status is within a safe range; and whether any abnormal situations affecting task execution or safety have occurred.

[0089] Step B20: If the abnormal situation exists, trigger the corresponding security intervention operation according to the abnormality type of the abnormal situation.

[0090] In this embodiment, an execution abnormality is determined to have occurred when any of the following situations are detected: motion deviation exceeds a preset threshold, motion state changes abruptly or becomes unstable, actuator operation is abnormal, or environmental changes cause the original motion path to become unsafe.

[0091] In one feasible implementation, abnormal situations are classified and then matched with a preset abnormality type and a safety intervention operation rule library. The abnormality types are divided into mild abnormality, moderate abnormality, and severe abnormality. Mild abnormality does not involve exceeding the motion deviation limit, slight overload of the actuator, or slight packet loss in communication; moderate abnormality does not involve severe overload of the actuator, low threshold collision force triggering, speed and attitude approaching the safety boundary, or temporary environmental occlusion of the target; severe abnormality does not involve high threshold collision force triggering, actuator stalling, temperature exceeding the limit, emergency triggering of safety constraints, joint limit triggering, or emergency stop signal input.

[0092] Retrieve the corresponding safety intervention operation according to the type of anomaly. For minor anomalies, fine-tune the parameters to reduce the speed and acceleration of the current action and reduce the single-step displacement to maintain task execution. For moderate anomalies, pause the execution of the current short-term action block, re-plan the local action path of the area, and reduce the actuator load limit. For severe anomalies, stop all action output, cut off the power to the actuator / brake, retreat the robot to the nearest safe position, and issue an audible and visual alarm.

[0093] According to the matching intervention rules, standardized intervention instructions are sent to the robot controller, and the execution status is confirmed. The intervention operation is converted into instructions that the actuator can recognize. The intervention instructions are sent through a high-priority communication channel, requiring the robot controller to execute immediately and receive an intervention execution confirmation frame. After the intervention is executed, feedback data is collected to verify whether the abnormality has been alleviated. If it has not been alleviated, the intervention level is upgraded.

[0094] Step B30: Update the semantic conditions and / or the action path based on the security intervention operation.

[0095] In this embodiment, the target parameters and constraint parameters in the original semantic conditions are extracted; the parameters are adjusted according to the safety intervention operation requirements; the modified semantic conditions are verified to meet the robot hardware limits; and after no parameter conflicts are found, they are marked as effective semantic conditions.

[0096] For scenarios requiring updated action paths, the original action path in the abnormal area is marked as invalid, and the issuance of instructions for that part is stopped. Based on the updated semantic conditions, the condition representation generation, short-term action block modeling, and smooth splicing process are re-executed to generate a new action path that adapts to the intervention requirements. The new path is smoothly spliced ​​with the effective part of the original path to ensure no sudden changes in action. The updated action path is issued to the execution mechanism according to the control cycle sequence to resume task execution.

[0097] In one feasible implementation, step B10 may include steps B11 to B14: Step B11: Determine the motion deviation based on the expected motion state data and the actual motion state data. If the motion deviation is greater than or equal to a preset deviation threshold, then the abnormal situation is confirmed to exist.

[0098] In this implementation, target parameters for the corresponding cycle, including position, velocity, and attitude, are extracted from the currently executed motion path to ensure consistency between the data dimensions and the actual feedback. The actual position, velocity, and attitude for the same cycle are acquired using robot sensors to standardize the data format. The two types of data are converted to the same coordinate system to avoid calculation errors caused by dimensional mismatch. The Euclidean distance between the actual and desired positions is calculated to obtain the position deviation; the absolute difference between the actual and desired velocities is taken to obtain the velocity deviation; and the absolute value of the angle difference between the actual and desired Euler angles is calculated to obtain the attitude deviation. A preset deviation threshold is obtained. If the motion deviation in any dimension is greater than or equal to the corresponding threshold, and this state persists for more than or equal to a preset number of control cycles, an abnormal situation is confirmed.

[0099] Step B12: Obtain the rate of change of motion parameters in adjacent cycles and the variance of motion parameters in a preset cycle. If the rate of change is greater than or equal to a preset rate of change threshold, and / or the variance of ...

[0100] In this embodiment, adjacent cycles are two consecutive control cycles, and the preset cycle is a sliding window of fixed length. Motion parameters such as velocity, acceleration, and joint angles are retrieved from historical feedback data within the window. The rate of change of motion parameters in adjacent cycles is calculated as |current cycle parameter value - previous cycle parameter value| / control cycle duration. A preset rate of change threshold is obtained. If the calculated rate of change is greater than or equal to the threshold, it is marked as an abnormal rate of change. The variance of the motion parameter sequence within the preset cycle, such as the velocity values ​​of 10 cycles, is calculated, and a preset fluctuation variance threshold is obtained. If the calculated fluctuation variance is greater than or equal to the threshold, it is marked as an abnormal fluctuation variance. When either the rate of change abnormality or the fluctuation variance abnormality condition is met, an anomaly is confirmed.

[0101] Step B13: Compare the actuator operating data of the robot with the corresponding normal operating parameter range. If the actuator operating data is not within the normal operating parameter range, the abnormal situation is confirmed.

[0102] In this embodiment, real-time operating data of each actuator is read according to the control cycle, including load rate, temperature, output torque / clamping force, speed, etc. The normal range for the corresponding actuator model is read from a preset actuator parameter library; the range can be dynamically adjusted according to the task type. The real-time operating data of each actuator is compared with the corresponding normal range. If the data is less than the lower limit of the range or greater than the upper limit, it is marked as the parameter being out of bounds. If any parameter of any actuator is out of bounds for a continuous period of ≥ a preset number of control cycles, an abnormal situation is confirmed, and the abnormal actuator number, the out-of-bounds parameter, and the difference exceeding the range are recorded. Step B14: If the distance between a path point and an obstacle point is less than a preset minimum safe distance, then the abnormal situation is considered to exist.

[0103] In this embodiment, the three-dimensional coordinates of the path points to be executed or currently being executed are obtained from the currently executing action path. The environment is scanned in real time using vision, LiDAR, and obstacle avoidance sensors to extract the three-dimensional coordinates of obstacle outlines, or the center coordinates and dimensions of the obstacle bounding box. For each path point, the shortest Euclidean distance to all obstacle points is calculated; if it is an obstacle bounding box, the shortest distance from the path point to the bounding box is calculated. A preset minimum safe distance threshold is obtained. If the shortest distance between a path point and an obstacle point is less than this threshold, an abnormal situation is confirmed, and the coordinates of the dangerous path point, the obstacle coordinates, and the actual distance are recorded.

[0104] This application provides a robot control device based on semantic-action integration, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the robot control method based on semantic-action integration in Embodiment 1 above.

[0105] The following is for reference. Figure 3 This document illustrates a structural schematic diagram of a robot control device based on semantic-action integration suitable for implementing embodiments of this application. The robot control device based on semantic-action integration in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, personal digital assistants (PDAs), tablet computers (PADs), in-vehicle terminals (e.g., in-vehicle navigation terminals), robots, unmanned vehicles, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The robot control device based on semantic action integration shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0106] like Figure 3 As shown, a semantic-action integrated robot control device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1002 or programs loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the semantic-action integrated robot control device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the semantic-motor integrated robot control device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a semantic-motor integrated robot control device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0107] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0108] The semantic-action integrated robot control device provided in this application, employing the semantic-action integrated robot control method described in the above embodiments, can solve the technical problem of how to reduce the response time of the visual language model reasoning process and improve reasoning efficiency. Compared with the prior art, the beneficial effects of the semantic-action integrated robot control device provided in this application are the same as those of the semantic-action integrated robot control method provided in the above embodiments, and other technical features in this semantic-action integrated robot control device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0109] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0110] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0111] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the robot control method based on semantic-action integration in the above embodiments.

[0112] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0113] The aforementioned computer-readable storage medium may be included in a semantic-action integrated robot control device; or it may exist independently and not assembled into the semantic-action integrated robot control device. The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the semantic-action integrated robot control device, the semantic-action integrated robot control device: performs semantic parsing and multimodal fusion of environmental information and language instructions based on a preset multimodal model to generate semantic conditions; generates action paths based on the semantic conditions and robot state information according to a preset control cycle; and sends the action paths to the robot, controlling the robot to perform corresponding actions based on the action paths.

[0114] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the client computer, partially on the client computer, as a standalone software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the client computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0117] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described semantic-action integrated robot control method. This addresses the technical problem of reducing the response time of the visual language model reasoning process and improving reasoning efficiency. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the semantic-action integrated robot control method provided in the above embodiments, and will not be repeated here.

[0118] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A robot control method based on semantic-action integration, characterized in that, The robot control method based on semantic-action integration includes: Based on a pre-defined multimodal model, semantic parsing and multimodal fusion are performed on environmental information and language instructions to generate semantic conditions; Based on a preset control cycle, an action path is generated according to the semantic conditions and robot state information; The motion path is sent to the robot, and the robot is controlled to perform the corresponding action based on the motion path.

2. The robot control method based on semantic-action integration as described in claim 1, characterized in that, Before the step of generating a motion path based on the semantic conditions and robot state information according to the preset control cycle, the robot control method based on semantic-motion integration further includes: When one or more of the following are detected: task execution deviation, environmental change, or robot state abnormality, the preset semantic update triggering conditions are confirmed to be met, and a semantic update signal is generated. The semantic conditions are updated based on the semantic update signal to obtain the updated semantic conditions.

3. The robot control method based on semantic-action integration as described in claim 2, characterized in that, The step of confirming that the preset semantic update triggering condition is met when one or more of the following are detected: task execution deviation, environmental change, or robot state abnormality. Based on the current environmental information and the environmental semantic information in the previous semantic cycle, determine whether there are changes in the target object, changes in the environmental structure, and / or changes in the passable area. If so, determine that the environmental changes exist. Based on the robot's execution feedback information, it is determined whether the actual motion result deviates from the expectation, the success rate of motion generation decreases, and / or there is a safety intervention operation in the motion path. If so, it is determined that there is a deviation in the task execution. When a safety constraint is triggered, the actuator state becomes abnormal, and / or the speed or attitude reaches a preset safety boundary threshold, it is determined that the robot state is abnormal.

4. The robot control method based on semantic-action integration as described in claim 1, characterized in that, The steps for generating semantic conditions by semantic parsing and multimodal fusion of environmental information and language instructions based on a preset multimodal model include: Extract basic visual features and task semantic features from the environmental information, and associate the basic visual features and the task semantic features to obtain visual semantic features; Semantic parsing of natural language instructions yields linguistic semantic features; The semantic conditions are obtained by weighted fusion and vectorization of the linguistic semantic features and the visual semantic features.

5. The robot control method based on semantic-action integration as described in claim 1, characterized in that, The step of generating a motion path based on a preset control cycle, semantic conditions, and robot state information includes: When the control cycle is reached, the robot state information and the semantic conditions are combined and encoded to obtain a conditional representation; Based on the condition representation and the preset time window, a short-term action change trend model is performed to obtain a short-term action block; For short-duration action blocks generated in adjacent time windows, smoothing and action block splicing are performed to determine the resulting continuous action sequence as the action path.

6. The robot control method based on semantic-action integration as described in claim 5, characterized in that, The step of modeling short-term action change trends based on the condition representation and a preset time window to obtain short-term action blocks includes: The condition representation is matched with a preset trend mapping rule library, and the movement trend corresponding to the condition representation is determined based on the target trend mapping rule obtained from the match. Based on a preset time-series generation model, the motion trend is feature extracted and encoded to obtain a trend feature vector; According to the time series pattern in the time series generation model, the trend feature vector is mapped to a time series parameter sequence; The time sequence parameter sequence is divided into sub-sequences based on the time window, and the sub-sequences are determined as the short-time action blocks.

7. The robot control method based on semantic-action integration as described in claim 1, characterized in that, The robot control method based on semantic-action integration also includes: Obtain feedback information from the robot during the execution of the action path, and determine whether there are any abnormal situations based on the feedback information; If the aforementioned abnormal situation exists, the corresponding security intervention operation will be triggered according to the abnormality type of the abnormal situation; The semantic conditions and / or the action path are updated based on the security intervention operation.

8. The robot control method based on semantic-action integration as described in claim 7, characterized in that, The step of obtaining feedback information from the robot during the execution of the motion path and determining whether there are any abnormalities based on the feedback information includes: The motion deviation is determined based on the expected motion state data and the actual motion state data. If the motion deviation is greater than or equal to a preset deviation threshold, the abnormal situation is confirmed to exist. Obtain the rate of change of motion parameters in adjacent cycles and the variance of motion parameters in a preset cycle. If the rate of change is greater than or equal to a preset rate of change threshold, and / or the variance of ... The actuator operating data of the robot is compared with the corresponding normal operating parameter range. If the actuator operating data is not within the normal operating parameter range, the abnormal situation is confirmed. If the distance between a path point and an obstacle point is less than the preset minimum safe distance, then the abnormal situation is considered to exist.

9. A robot control device based on semantic-action integration, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot control method based on semantic-action integration as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the robot control method based on semantic-action integration as described in any one of claims 1 to 8.

Images