A robot control method and system based on artificial intelligence

Abstract

This application provides an artificial intelligence-based robot control method and system, relating to the field of intelligent robots, and solves the technical problems of low execution efficiency and weak safety of existing robots in unfamiliar environments. The method includes: collecting visual data, depth data, 3D point cloud data, and audio data of the robot's environment; processing each modal data to obtain multimodal feature vectors and event labels; constructing dynamic entities representing environmental elements based on the multimodal feature vectors; assigning a multidimensional semantic vector to each dynamic entity to generate a context graph describing the relationships between entities; performing graph similarity matching between the context graph and a pre-constructed semantic memory network; performing inference based on the matching results to generate a hierarchical task tree; executing atomic action sequences in the hierarchical task tree; and adjusting the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequences of the dynamic entities and the context graph.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robots, specifically a robot control method and system based on artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, robots are expected to operate autonomously, safely, and efficiently in increasingly complex unstructured environments such as industrial manufacturing, home services, and emergency rescue. In existing technologies, robot control methods mainly follow the classic architecture of perception, planning, and execution, and representative methods with corresponding limitations exist at different levels.

[0003] At the level of environmental perception and understanding, existing robotic technologies primarily rely on mathematical modeling centered on recognition and segmentation, which has several limitations. First, while deep learning-based recognition technologies can determine object categories, this only completes descriptive semantic extraction. The functional semantics required for robot task execution are difficult to obtain. Whether relying on massive labeled datasets with poor scalability or on computationally expensive physical simulations that depend on precise parameters, neither approach is practical in open environments. Second, current mainstream scene representations (such as semantic point cloud maps) are essentially static snapshots of the environment, failing to explicitly and structurally model the spatiotemporal and logical relationships between objects and between objects and events, thus unable to support task planning that requires complex reasoning. Therefore, existing perception technologies enable robots to "see" but not fully "understand," and the world models they construct are information-poor and rigid, restricting robots from achieving truly autonomous behavior in unstructured environments.

[0004] At the task planning and decision-making level, mainstream methods can be divided into two categories: one is based on pre-programming or teach-and-reproduce, where the planned paths and action sequences are fixed and cannot adapt to any changes in the environment or task, exhibiting extremely poor flexibility; the other is based on online motion planning using search or optimization algorithms, which can handle some dynamics but is zero-shot planning, requiring recalculation each time a new scenario is encountered, resulting in low decision-making efficiency and heavy reliance on accurate environmental models, performing poorly when the model is uncertain or partially observable. Although AI methods such as reinforcement learning have been introduced for policy learning, they typically require massive amounts of simulation or real-world interaction data for training, and the learned policies are black boxes with limited generalization ability, making it difficult to accumulate and reuse cross-task experience knowledge.

[0005] At the execution control and adaptability level, existing systems mostly employ open-loop execution or closed-loop based on simple error feedback (such as PID control). The former relies entirely on the accuracy of the planning, while the latter mainly adjusts low-level motion parameters. Both lack the ability to perform high-level semantic monitoring and evaluation of execution performance at the task level. When unexpected disturbances or planning premises fail during execution, the system often fails to intelligently adjust its strategy, either mechanically executing and causing failure or danger, or directly reporting an error and shutting down. Summary of the Invention

[0006] This application provides a robot control method and system based on artificial intelligence, which solves the technical problems of poor adaptability, slow decision-making and weak safety of robots in unfamiliar, dynamic and unstructured complex environments due to shallow understanding of the environment, low efficiency of task planning and lack of intelligent adjustment ability in the execution process.

[0007] To achieve the above objectives, this application adopts the following technical solution: Firstly, an artificial intelligence-based robot control method is provided, including: The robot collects visual data, depth data, 3D point cloud data, and audio data of its environment, and processes the data in parallel to obtain multimodal feature vectors and event labels. The multimodal feature vectors are spatiotemporally aligned and associated to construct dynamic entities representing environmental elements. Each dynamic entity is given a multidimensional semantic vector containing physical attributes, functional attributes, and event labels to generate a contextual graph describing the relationships between entities. The context graph and the pre-built semantic memory network are matched for graph similarity. Reasoning is performed based on the matching results to generate a hierarchical task tree. The semantic memory network is built based on the incremental experience of historical task execution and is used to store and associate task planning and results in different contexts. The hierarchical task tree is used to guide the robot's action execution, including parameterized atomic action sequences. Execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence of the dynamic entity and the context graph.

[0008] Based on the above technical solutions, the AI-based robot control method provided in this application achieves deep semantic fusion and structured understanding of multimodal environmental information by constructing dynamic entity and context graphs, enabling the robot to leap from low-level signal processing to high-level relational cognition. Then, a semantic memory network built based on incremental historical experience is introduced and queried, allowing the robot to perform analogical reasoning and quickly generate task plans using past successful or failed experiences, improving decision-making efficiency and adaptability in unfamiliar environments. Finally, by real-time monitoring of semantic state change sequences and comparing them with the context graph, the robot can not only execute actions based on pre-planned actions but also autonomously and dynamically adjust parameters and strategies during task execution. This ensures that the robot possesses speed, safety, and robustness in task completion when facing uncertainty, achieving a crucial leap from perceptual intelligence to cognitive decision-making intelligence.

[0009] Furthermore, the multimodal feature vector includes an appearance feature vector, a geometric feature vector, and a structural feature vector, and the generation process of the multimodal feature vector and the event label includes: The visual data and the depth data are registered, and the registered image is input into a pre-trained convolutional neural network for feature extraction to obtain an appearance feature vector; wherein, the visual data is color image data acquired by an RGB camera; Based on the depth data, the three-dimensional coordinates corresponding to each pixel in the registered image data are calculated, and the length and direction of the geometric axes of the three-dimensional point cloud are extracted by principal component analysis to obtain the geometric feature vector; the depth data is data collected by a depth camera or lidar to characterize the distance from the object surface to the sensor. For the three-dimensional point cloud data, a random sampling consensus algorithm is used to perform plane fitting, and the normal vectors of no less than three maximum planes and the distance from the plane to the origin are extracted to jointly form the structural feature vector; The audio data is first pre-emphasized and windowed in frames, and then the Mel frequency cepstral coefficients of each frame are calculated to obtain a feature sequence. The feature sequence is input into a pre-trained recurrent neural network, which outputs an event classification probability distribution. The category k with the largest probability value in the probability distribution is taken as the event label for the current time period. The event label includes, but is not limited to, human voices, motor sounds, collision sounds, and alarm sounds.

[0010] Furthermore, the step of spatiotemporally aligning and associating the multimodal feature vectors to construct dynamic entities representing environmental elements includes: The visual data is input into a pre-trained target detection neural network to obtain a two-dimensional object detection bounding box of a predefined target, and the bounding box is bound to the appearance feature vector of the corresponding predefined target to obtain a first binding pair; The three-dimensional point cloud data is clustered using a clustering algorithm to obtain three-dimensional point cloud clusters. The three-dimensional point cloud clusters are then bound to the geometric feature vectors and structural feature vectors at corresponding locations to obtain second and third binding pairs. Based on the spatial alignment and feature correlation determination of each binding pair, matching pairs determined to originate from the same environmental physical element are obtained. The two-dimensional object detection bounding boxes in the matching pair are fused with the three-dimensional point cloud clusters, and the fusion result is associated with the multimodal feature vectors in the corresponding first binding pair, second binding pair, and third binding pair to jointly construct a dynamic entity.

[0011] Furthermore, the spatial alignment and feature correlation determination based on each of the binding pairs includes: The cosine similarity between the appearance feature vector in the first binding pair and the appearance feature vector extracted from the image region corresponding to the point cloud cluster is calculated as the appearance feature similarity. The cosine similarity between the geometric feature vector in the second binding pair and the geometric feature vector extracted from the depth map region corresponding to the bounding box is calculated as the geometric feature similarity. The cosine similarity between the structural feature vector in the third binding pair and the structural feature vector extracted from the depth map region corresponding to the bounding box is calculated as the structural feature similarity. Project the 3D point cloud clusters in the second binding pair onto the image plane, and calculate the intersection-union ratio of the projected region with the 2D object detection bounding boxes in the first binding pair; If the intersection-union ratio is greater than a preset spatial alignment threshold, and the weighted sum of the geometric feature similarity, the appearance feature similarity, and the structural feature similarity is greater than a preset feature similarity threshold, then it is determined that the two-dimensional object detection bounding box and the three-dimensional point cloud cluster originate from the same environmental physical element, forming a matching pair.

[0012] Furthermore, the step of assigning a multi-dimensional semantic vector containing physical attributes, functional attributes, and event labels to each dynamic entity to generate a contextual graph describing the relationships between entities includes: The physical attribute vector of the dynamic entity is derived from the geometric feature vector and the structural feature vector; The apparent feature vector and physical attribute vector of the dynamic entity are input into a pre-trained neural network for target recognition, and the functional attribute vector of the dynamic entity is output; wherein, the functional attributes include the graspability score, pushability score, pressability score, rotatability score, standing ability score, obstacle level, communication cost, and role label of the dynamic entity. The event tag vector of the dynamic entity is obtained based on the one-hot encoding of the event tag; Each dynamic entity is treated as a graph node, and the node feature vector of the graph node is composed of the physical attribute vector, the functional attribute vector, and the event label vector. The spatial relationship between any two graph nodes is calculated based on the node feature vectors, and the temporal relationship between any two graph nodes is obtained according to the event occurrence time; wherein, the spatial relationship includes, but is not limited to, distance, azimuth angle, elevation angle, and containment relationship; The current context graph is constructed using a set of graph nodes and a set of labeled edges; wherein the labels of the edge set include the node IDs of the two graph nodes that make up the edge, the spatial relationship between the two graph nodes, and the temporal relationship.

[0013] Furthermore, the construction process of the pre-trained neural network includes: Collect operation records of the robot when performing tasks in different scenarios. Each record includes RGB-D image sequence, synchronized 3D point cloud, robotic arm joint status and task success / failure annotations to obtain a historical record library; Using the method for constructing dynamic entities, dynamic entity samples are extracted from the historical record database. Each sample contains an appearance feature vector, a physical attribute vector, and a manually labeled functional attribute vector as sample labels, thus obtaining a dataset. The dataset is divided into a training set and a validation set according to a preset ratio; A multi-layer feedforward neural network is constructed, with concatenated apparent feature vectors and physical attribute vectors as inputs and labeled functional attribute vectors as supervision targets. A combination of mean squared error loss function and cross-entropy loss function is used as the training target. The network is trained on the training set through backpropagation algorithm and optimizer until the prediction error on the validation set converges, thus obtaining the pre-trained neural network.

[0014] Furthermore, the construction process of the semantic memory network includes: A complete task execution process is encapsulated into a memory unit, which includes a context diagram at the start of the task, a hierarchical task tree, a semantic state change sequence, and the task result. When a new memory unit is added, the initial context graph of the new memory unit is matched with the initial context graph of the existing memory unit. If there are node pairs in the context graphs of the new memory unit and the existing memory unit with a semantic similarity greater than a preset threshold, the node pairs are mapped to the same abstract semantic category node. The newly added memory units are connected to the abstract semantic category nodes. At the same time, edges are established between the newly added memory units and the existing memory units according to the association relationship of the hierarchical task tree in the existing memory units to obtain the semantic memory network. The association relationship includes, but is not limited to, inclusion, comparison and causality.

[0015] Furthermore, the formula for calculating the semantic similarity is as follows: Where FuncVec is the functional attribute vector, PhyVec is the physical attribute vector, CosineSim is the cosine similarity calculation function, a and b are node indices, and ω1 and ω2 are preset weight coefficients with ω1+ω2=1.

[0016] Further, the step of performing graph similarity matching between the context graph and the pre-constructed semantic memory network includes: The current context graph is matched with the context graph at the beginning of each memory unit in the memory network. The graph similarity score is calculated by weighted summation of node similarity and edge similarity between context graphs. Based on a preset similarity threshold, memory units with graph similarity scores greater than the preset similarity threshold are selected as candidate experience sets for the current decision.

[0017] Furthermore, the step of generating a hierarchical task tree based on the matching results includes: If the candidate experience set is not empty, the hierarchical task tree stored in the memory unit with the highest graph similarity is used as the template of the current hierarchical task tree, and the atomic action parameters of the current hierarchical task tree are adjusted according to the node differences between the current context graph and the context graph stored in the memory unit with the highest graph similarity. If the candidate experience set is empty, then with the current task objective as the root node, traverse the entities in the current context graph whose functional attributes satisfy the current task objective, and generate a sub-objective sequence based on the physical attributes and spatial relationships of the entities through a predefined rule base; then decompose each sub-objective into executable parameterized atomic actions in the robot motion planning library to obtain the hierarchical task tree of the current task.

[0018] Furthermore, adjusting the atomic action parameters of the current hierarchical task tree based on the node differences between the current context graph and the context graph stored in the memory unit with the highest graph similarity includes: The context graph stored in the memory unit with the highest graph similarity is marked as a template context graph, and the hierarchical task tree stored in the memory unit with the highest graph similarity is marked as a template task tree; Extract the dynamic entities that correspond to the current context graph and the template context graph but have different node feature vectors to obtain the differential dynamic entities; Based on the differences in the physical attribute vectors of the differential dynamic entities, the atomic action parameters of the template task tree applied to the differential dynamic entities are scaled proportionally. Based on the differences in the spatial relationship vectors of the differential dynamic entities, coordinate transformation is performed on the target pose parameters of the moving atomic actions in the template task tree, so that the transformed atomic actions are adapted to the actual positions of the differential dynamic entities.

[0019] Furthermore, the step of generating a sub-target sequence based on the physical attributes and spatial relationships of the entity using a predefined rule base includes: Match the current task target with the entities obtained through traversal that meet the functional attribute conditions to determine the target entity for task execution; Combining the functional attribute vector and physical attribute vector of the target entity, an operation rule is selected from a predefined rule base; the operation rule is used to instantiate the target entity into a sub-target. Based on the aforementioned operation rules, the target entity is instantiated as a main operation sub-target; Based on the spatial relationship between the target entity and other entities, a preliminary sub-target is inserted; wherein, the preliminary sub-target is an auxiliary sub-target that must be completed in advance to ensure that the main operation sub-target can be executed, including but not limited to: moving to a preset range of the target entity, removing obstacles on the path, and adjusting the robot end effector posture; The main operation sub-objective and the preliminary object are sorted according to logical dependencies and spatial reachability to obtain a sub-objective sequence.

[0020] Furthermore, the predefined rule base is organized hierarchically by scope, including but not limited to: Object operation rules: Define executable operations for entities with specific functional attributes, in the form of: IF (target entity.functional attribute.operation feasibility score > threshold_operation) AND (target entity.physical attribute.dimension < robot capability limit) THEN (add sub-goal: operation type (target entity)). For example: IF (target entity.graspability > 0.7) AND (target entity.volume < robot gripping volume) THEN (add sub-goal: grip (target entity)); Spatial Relationship Rules: Used to handle the relative positional relationships between entities and generate preliminary sub-objectives. The form is: IF(spatial relationship between target entity and robot.distance>threshold_close) THEN(insert preliminary sub-objective: move to (near target entity)); or IF(obstacle entity blocks the path to target entity) THEN(insert preliminary sub-objective: avoid obstacle (obstacle entity) or move away (obstacle entity)).

[0021] The rule matching process follows a specific priority; for example, spatial relationship rules are executed before object operation rules to ensure unobstructed paths. When generating a sub-target sequence, the system recursively applies the matching rules based on the current context graph and sorts the instantiated sub-targets according to logical order and spatial reachability to form an ordered sub-target sequence.

[0022] Furthermore, the semantic state change sequence is obtained as follows: When executing a hierarchical task tree, a set of semantic state changes is preset for each atomic action, which includes the target entity, the attribute index to be changed, and the expected value of the change. After the atomic action is executed, the semantic vector of the acting entity is sampled at a fixed frequency to obtain the attribute change value of the acting entity; wherein, the acting entity is the entity that the atomic action affects; The sequence of recorded attribute change values ​​over time is compared with the expected change value to obtain the actual semantic state change fragment corresponding to the atomic action; By splicing together the actual semantic state change fragments generated by all atomic actions during the entire task execution process in chronological order, a complete semantic state change sequence is obtained.

[0023] Further, adjusting the action parameters of atomic actions and the task execution path of the hierarchical task tree based on the semantic state change sequence of the dynamic entity and the context graph includes: When the trend of the attribute change value of the active entity is found to be consistent with the expected change value, but the rate of change is lower than the preset rate value, the difference between the two is recorded as the error, and then the action parameter is adjusted using a proportional control law; the adjustment formula for the action parameter u(t) is: ; These are the baseline parameters for atomic actions. For proportional gain, The error is denoted as t, where t represents time. When it is detected that the sign of the error is always opposite to the expected value of the change within N consecutive detection cycles, or the integral of the absolute value of the error exceeds a preset threshold, the atomic action is determined to have failed. Then, graph similarity matching and reasoning are performed again based on the latest context graph to generate a new hierarchical task tree, thereby adjusting the task execution path; where N is a positive integer.

[0024] Furthermore, the method also includes: after the task segment ends, encapsulating the initial context graph, the hierarchical task tree, the semantic state change sequence, and the final execution result into a new memory unit, and integrating the new memory unit into the semantic memory network in a structured manner to enhance the understanding and reasoning ability for subsequent task decisions.

[0025] Secondly, this application provides an artificial intelligence-based robot control system, including: a data acquisition and processing module, a context graph construction module, a task planning module, and a task execution module; wherein, The data acquisition and processing module is used to acquire visual data, depth data, 3D point cloud data and audio data of the robot's environment, and to process the data of each modality in parallel to obtain multimodal feature vectors and event labels. The context graph construction module is used to perform spatiotemporal alignment and association of the multimodal feature vectors, construct dynamic entities representing environmental elements, and assign a multidimensional semantic vector containing physical attributes, functional attributes and event labels to each dynamic entity to generate a context graph describing the relationships between entities. The task planning module is used to perform graph similarity matching between the context graph and the pre-built semantic memory network, and to perform reasoning based on the matching results to generate a hierarchical task tree; wherein, the semantic memory network is built based on historical task execution experience increments and is used to store and associate task planning and results under different contexts; the hierarchical task tree is used to guide robot action execution, including parameterized atomic action sequences; The task execution module is used to execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence of the dynamic entity and the context graph.

[0026] Compared with the prior art, the beneficial effects of this application are: At the perception and understanding level, this application constructs dynamic entities that integrate visual, geometric, and structural features through parallel processing of multimodal data, spatiotemporal alignment, and feature correlation determination. These entities are then endowed with rich semantics, including physical, functional, and event labels, ultimately generating a contextual graph describing the relationships between entities. This process enables robots to transcend traditional object recognition and localization, achieving semantic and relational understanding of the environment. It enhances the robustness and accuracy of perception in unstructured, dynamic environments, as well as the ability to recognize the potential uses of objects, laying a solid foundation for subsequent intelligent decision-making. At the planning and decision-making level, this application constructs and queries a semantic memory network based on incremental accumulation of historical experience, and utilizes graph similarity matching for analogical reasoning. This enables the robot to quickly recall or adapt previously successful task plans, thereby significantly reducing the computational overhead and trial-and-error costs of planning from scratch when facing new scenarios. When direct experience is lacking, the system can autonomously generate hierarchical task trees based on reasoning about entity functional attributes and spatial relationships through predefined rules. This dual-mode decision-making mechanism, combining experience-driven and reasoning-based generation, enhances the robot's task planning efficiency, success rate, and adaptability in completely unfamiliar environments. At the execution and learning level, this application introduces a real-time monitoring and closed-loop adjustment mechanism based on semantic state change sequences. The robot not only executes preset actions but also continuously verifies whether the environmental semantic changes triggered by its actions meet expectations. Based on this, it can fine-tune action parameters online or decisively trigger task replanning, improving the compliance, safety, and ultimate success rate of task execution. Simultaneously, the experience gained after task completion is structured, encapsulated, and integrated into a semantic memory network, enabling the entire system to possess continuous lifelong learning and evolution capabilities. This provides a complete and advanced technical solution for addressing the challenges of safe, efficient, and autonomous robot operation in complex real-world scenarios. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A system architecture diagram of an artificial intelligence-based robot control system provided for embodiments of this application; Figure 2 A flowchart illustrating an artificial intelligence-based robot control method provided in this application embodiment; Figure 3 A flowchart illustrating another AI-based robot control method provided in this application embodiment; Figure 4 A flowchart illustrating another AI-based robot control method provided in this application embodiment; Figure 5 This is a flowchart illustrating another AI-based robot control method provided in an embodiment of this application. Detailed Implementation

[0029] The robot control method based on artificial intelligence provided in this application embodiment can be applied to, for example... Figure 1 In an AI-based robot control system, as shown, Figure 1 As shown, the system includes: a data acquisition and processing module, a scenario graph construction module, a task planning module, and a task execution module; among which, The data acquisition and processing module is used to collect visual data, depth data, 3D point cloud data and audio data of the robot's environment, and to process the data of each modality in parallel to obtain multimodal feature vectors and event labels. The context graph construction module is used to align and associate multimodal feature vectors in time and space, construct dynamic entities representing environmental elements, and assign a multidimensional semantic vector containing physical attributes, functional attributes and event labels to each dynamic entity, thereby generating a context graph describing the relationships between entities. The task planning module is used to perform graph similarity matching between the context graph and the pre-built semantic memory network, and to perform reasoning based on the matching results to generate a hierarchical task tree. The semantic memory network is built incrementally based on historical task execution experience and is used to store and associate task planning and results under different contexts. The hierarchical task tree is used to guide the robot's action execution, including parameterized atomic action sequences. The task execution module is used to execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence and context graph of the dynamic entity.

[0030] To address the technical problems in existing technologies where robots exhibit shallow environmental understanding, low task planning efficiency, and a lack of intelligent adjustment capabilities during execution in unfamiliar, dynamic, and unstructured complex environments, resulting in poor robot adaptability, slow decision-making speed, and weak operational safety, this application provides an artificial intelligence-based robot control method, which includes: The robot collects visual data, depth data, 3D point cloud data, and audio data of its environment, and processes the data in parallel to obtain multimodal feature vectors and event labels. Multimodal feature vectors are spatiotemporally aligned and associated to construct dynamic entities representing environmental elements. Each dynamic entity is given a multidimensional semantic vector containing physical attributes, functional attributes, and event labels to generate a contextual graph describing the relationships between entities. The context graph and the pre-built semantic memory network are matched for graph similarity. Reasoning is performed based on the matching results to generate a hierarchical task tree. The semantic memory network is built based on the incremental experience of historical task execution and is used to store and associate task planning and results in different contexts. The hierarchical task tree is used to guide the robot's action execution, including parameterized atomic action sequences. Execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence and context graph of the dynamic entity.

[0031] Based on this, the method enables robots to achieve deep semantic understanding and structured cognition of the environment, improves decision-making efficiency in unfamiliar environments by leveraging historical experience, and enables robots to balance reaction speed, operational safety and task completion robustness when facing environmental uncertainties through dynamic adjustments in the execution process, thus achieving a leap from perceptual intelligence to cognitive decision-making intelligence.

[0032] This application provides an artificial intelligence-based robot control method, comprising: S1. Collect visual data, depth data, 3D point cloud data, and audio data of the robot's environment, and process the data of each modality in parallel to obtain multimodal feature vectors and event labels.

[0033] Among them, visual data can be acquired by red-green-blue (RGB) cameras, depth data can be acquired by depth cameras or LiDAR, 3D point cloud data can be acquired by LiDAR or 3D vision sensors, and audio data can be acquired by microphones; and the extracted event tags are category information that identifies specific event types in the robot's environment, and are the classification results of event content reflected by modal data such as audio in the environment.

[0034] In some implementations, a multi-threaded processing architecture can be used to process the data of each modality in parallel. For example, visual feature vectors can be extracted from visual data using a convolutional neural network, depth feature vectors can be extracted from depth data using feature mapping and dimensionality reduction, point cloud feature vectors can be extracted from 3D point cloud data using voxelization and feature encoding, and audio feature vectors can be extracted from audio data using time-frequency domain analysis. All these feature vectors together constitute a multimodal feature vector. At the same time, the audio data can be combined with a feature classification model to identify event types and output the corresponding event labels.

[0035] It should be noted that event tags can quickly identify key event information in the environment for robots, enabling them to perceive dynamic events occurring in the environment. This provides event-dimensional references for subsequent environmental understanding and task decision-making, assisting robots in judging the environmental state. For example, if collision-related voiceprint features are detected in audio data, and a collision sound event tag is output, the robot can perceive the occurrence of a collision event in the environment and make corresponding risk assessments for action planning.

[0036] S2. Spatiotemporally align and associate the multimodal feature vectors to construct dynamic entities representing environmental elements, and assign each dynamic entity a multidimensional semantic vector containing physical attributes, functional attributes, and event labels to generate a contextual graph describing the relationships between entities.

[0037] Spatiotemporal alignment and association involve calibrating different modal data to the same time reference in the time dimension and matching them to the same spatial coordinate system in the spatial dimension. At the same time, it establishes the correlation between the feature vectors of each modality and determines the same environmental target corresponding to the feature vectors. Dynamic entities are digital and model-based representations of various physical elements in the robot's environment. They are digital models of environmental elements that integrate multimodal features. Physical attributes are the physical characteristics of the environmental physical elements corresponding to the dynamic entities. They typically include size, shape, material, spatial coordinates, motion state, etc. Functional attributes are the functional characteristics of the environmental physical elements corresponding to the dynamic entities that can be used by the robot or that the robot possesses. They typically include operability type, operability, and cooperation functions with other entities.

[0038] In some implementations, time alignment can be achieved by synchronizing timestamps to calibrate the feature vectors of each modality to a unified time axis, while spatial alignment can be achieved by mapping the feature vectors of each modality to a unified spatial coordinate system using coordinate transformation and registration. Feature association can be achieved by establishing the correspondence between feature vectors of different modalities through similarity calculation. Then, based on the associated multimodal feature vectors, dynamic entities corresponding to environmental elements are aggregated. For each dynamic entity, physical attributes, functional attributes, and event tags are fused to generate a multidimensional semantic vector. Each dynamic entity is then used as a node in a graph, and the spatiotemporal and logical relationships between entities are used as edges of the graph, thereby constructing a context graph.

[0039] It should be noted that context maps can present the characteristics of each element in the environment and the relationships between them in a structured way, allowing robots to understand the semantic structure of the environment as a whole. This breaks through the limitations of traditional static environment modeling and provides a comprehensive and dynamic environmental cognition foundation for subsequent task planning.

[0040] For example, in an indoor environment, tables, chairs, and water cups can be constructed as dynamic entities, and each entity can be given physical attributes such as size and material, as well as functional attributes such as placement and grabbing. Semantic vectors can be generated by combining event tags in the environment, and then spatial relationships such as distance and orientation between entities can be used as edges to construct a contextual graph that reflects the relationships between indoor environmental elements.

[0041] S3. Perform graph similarity matching between the context graph and the pre-constructed semantic memory network, and perform reasoning based on the matching results to generate a hierarchical task tree.

[0042] Among them, the semantic memory network is built based on the incremental experience of historical task execution and is used to store and associate task planning and results in different contexts; the hierarchical task tree is used to guide the robot's action execution, including parameterized atomic action sequences.

[0043] In some implementations, the construction of a semantic memory network can begin by encapsulating a single, complete task execution process into an independent memory unit. Each memory unit contains the initial environmental context of the task execution, the task planning scheme, the execution process data, and the task result. Then, through incremental learning, new memory units are associated with existing memory units based on contextual similarity, forming a semantic memory network with hierarchy and association relationships. The construction of a hierarchical task tree can begin by determining the overall goal of the task as the root node, then decomposing the overall goal into several sub-goals, and further decomposing the sub-goals into executable basic actions. Specific execution parameters are configured for each basic action to form a parameterized sequence of atomic actions. The goals and atomic actions are combined according to logical hierarchical relationships to form a hierarchical task tree. The final hierarchical task tree can include the task root node, sub-goal nodes at all levels, atomic action nodes, as well as the logical execution relationships between nodes, parameter information of atomic actions, and the order of action execution.

[0044] It should be noted that semantic memory networks can realize the structured storage and efficient retrieval of robots' historical task experience, allowing robots to make quick decisions based on past experience when facing new environmental situations, reducing the cost of planning from scratch; hierarchical task trees can decompose complex overall tasks layer by layer, transforming them into atomic actions that robots can execute, providing robots with clear and orderly action execution guidance, making the execution of complex tasks more systematic.

[0045] For example, when the robot performs the task of retrieving a water cup, it retrieves memory units from similar indoor environments in the semantic memory network to decompose the overall goal of retrieving the water cup into three sub-goals: moving to the water cup position, grasping the water cup, and moving to the target position. Each sub-goal is further decomposed into atomic actions such as robotic arm movement and gripper opening and closing, and parameters are configured to form a hierarchical water cup retrieval task tree.

[0046] S4. Execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence and context diagram of the dynamic entity.

[0047] Among them, the semantic state change sequence is a continuous data sequence in which the semantic vector of the dynamic entity being acted upon changes over time during the execution of atomic actions by the robot, reflecting the dynamic changes of entity attributes during the execution of actions.

[0048] In some implementations, the atomic actions can be adjusted using feedback control. Based on the deviation between the semantic state change sequence and the expected state, the execution parameters of the atomic actions are corrected according to control rules such as proportional, integral, and differential. The task execution path of the hierarchical task tree can be adjusted by first judging the effect of the current action execution based on the real-time context graph and semantic state change sequence. If the action execution is determined to be unsuccessful or the current path cannot complete the task, the matching reasoning between the context graph and the semantic memory network is performed again, and the sub-target nodes and atomic action nodes of the task tree are deleted, added, or adjusted to form a new task execution path.

[0049] It should be noted that this step enables closed-loop intelligent adjustment during the robot's task execution process, allowing the robot to no longer mechanically execute preset actions, but to dynamically optimize action parameters and task paths based on real-time changes in the environment and the physical entity. This effectively addresses environmental interference and planning deviations during execution, improving the robot's success rate and operational safety in complex dynamic environments.

[0050] For example, when the robot performs the atomic action of grasping a water cup, it finds that the spatial position semantic state of the water cup deviates significantly from the expected value. Based on the deviation value, the robot adjusts the movement coordinate parameters of the robotic arm. If it finds that the water cup is obstructed, preventing the grasping action from being completed, the task path is replanned, adding a sub-target and atomic action to remove the obstruction, and adjusting the execution order of the original task tree.

[0051] Based on the above technical solutions, this application provides an artificial intelligence-based robot control method that achieves comprehensive perception of multi-dimensional environmental information through multimodal data acquisition and parallel processing. By constructing dynamic entities and contextual graphs through spatiotemporal alignment and association, it completes deep semantic fusion and structured understanding of the environment, enabling the robot to improve from low-level signal processing to high-level relational cognition. Through graph similarity matching and reasoning of semantic memory networks, the robot can reuse historical task experience, improving task planning efficiency and adaptability in unfamiliar environments. Through dynamic adjustment based on semantic state change sequences during execution, the robot possesses intelligent feedback and optimization capabilities at the task level, effectively coping with environmental uncertainties and balancing action execution reaction speed, operational safety, and task completion robustness. It also lays the foundation for continuous autonomous learning and capability evolution, enabling the robot to operate more autonomously, safely, and efficiently in unstructured complex environments.

[0052] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S1 can be implemented through the following S101, S102 and S103, which are explained in detail below: S101. Collect visual data, depth data, 3D point cloud data, and audio data of the robot's environment to complete the acquisition and preliminary processing of multi-source environmental data.

[0053] Among them, visual data is color image data obtained by the robot through visual perception of its environment; depth data is ranging data that represents the distance from the surface of an object in the environment to the sensor; 3D point cloud data is a set of 3D coordinate points on the surface of an object in the environment; and audio data is audio data collected by the robot from various sound signals in the environment. These four types of data enable the robot to have a comprehensive perception of its surrounding environment from different dimensions.

[0054] In some implementations, S101 may include the following operational steps: Dedicated sensing devices are used to accurately collect various types of data. At the same time, a unified timestamp synchronization module is configured for all collection devices to keep the multi-source data collected by different sensing devices synchronized in the time dimension. All types of raw data collected are stored in the robot's local data cache module, classified and archived according to data type for easy retrieval and processing later.

[0055] For example, in an industrial manufacturing scenario, the robot uses an RGB camera to collect color visual images of the production line and workpieces, uses a LiDAR to collect depth data and 3D point cloud data of the production line equipment and workpieces, and uses a microphone to collect audio data of motor operation and workpiece collisions in the production line. Under the control of the timestamp synchronization module, all devices complete the collection of corresponding data at the same time node, and various types of data are classified and stored in the cache module.

[0056] S102. Parallel feature extraction is performed on the collected visual data, depth data, and 3D point cloud data to obtain a multimodal feature vector composed of appearance feature vector, geometric feature vector, and structural feature vector.

[0057] Among them, the appearance feature vector is a vector that digitally represents the visual appearance of objects in the environment, the geometric feature vector is a vector that digitally represents the three-dimensional spatial geometric morphology of objects in the environment, and the structural feature vector is a vector that digitally represents the spatial structure and planar distribution of objects in the environment. The three types of feature vectors characterize the feature attributes of environmental elements from different dimensions and together constitute multimodal feature vectors.

[0058] In some implementations, a multi-threaded parallel processing architecture is used to extract features from the three types of data separately. Each thread runs independently and the data processing progress is synchronized in real time. The specific extraction process is as follows: The first step is to register the visual data and the depth data, mapping the pixel coordinates of the depth data to the pixel coordinate system of the visual data to achieve spatial matching of the two types of data. The registered fused image is then input into a convolutional neural network (CNN) pre-trained on a large-scale image dataset. Through layer-by-layer feature extraction of convolutional and pooling layers and feature mapping of fully connected layers, a high-dimensional appearance feature vector is output, which contains the appearance visual features of the object such as color, texture, and contour. The second step is to calculate the three-dimensional world coordinates of each pixel in the registered image data based on the conversion formula between pixel coordinates and depth values, thereby forming the preliminary shape of the three-dimensional point cloud data. Then, the principal component analysis (PCA) algorithm is used to extract features from the three-dimensional point cloud data. By calculating the covariance matrix of the point cloud data, the eigenvalues ​​and eigenvectors of the covariance matrix are solved, and the length and direction features of the geometric axes of the three-dimensional point cloud are extracted and digitized to form geometric feature vectors. The third step is to preprocess the 3D point cloud data, remove outliers and noise points, and then use the Random Sampling Consensus (RANSAC) algorithm to fit it to a plane. By iteratively sampling, the interior point data in the plane is selected, and the optimal plane model is fitted. At least three maximum planes are extracted from the 3D point cloud data in sequence, and the normal vector and the distance from the plane to the origin of the coordinate system are calculated for each plane. The normal vector parameters and distance parameters of all planes are concatenated in a preset order to form a structural feature vector.

[0059] It should be noted that using parallel processing for multimodal feature extraction can significantly improve data processing efficiency, avoid the time-consuming problem of single-threaded processing, and adapt to the real-time perception needs of robots in dynamic environments.

[0060] For example, after registering the visual data and depth data of the collected water cup, the data is input into a pre-trained convolutional neural network to extract the appearance features of the water cup, such as color, texture, and rim outline, and form an appearance feature vector. Based on the registered depth data, the three-dimensional coordinates of each pixel of the water cup are calculated. Principal component analysis is used to extract the geometric features of the water cup, such as height, rim diameter, and overall spatial orientation, and form a geometric feature vector. After denoising the three-dimensional point cloud data of the water cup, the three largest planes of the side, bottom, and rim of the cup are fitted using a random sampling consistency algorithm. The normal vector of each plane and its distance to the origin are calculated and then concatenated to form the structural feature vector of the water cup.

[0061] S103. Process the collected audio data and classify and identify events to obtain event labels that can characterize the types of events in the environment.

[0062] Among them, event tags are category information that identifies specific event types occurring in the robot's environment. They are obtained through feature analysis and classification of audio data, reflecting dynamic sound events in the environment and providing event-dimensional information support for the robot's environmental understanding.

[0063] In some implementations, S103 includes the following steps: The first step is to preprocess the acquired raw audio data. First, a pre-emphasis operation is performed to enhance the high-frequency components in the audio signal through a first-order high-pass filter to compensate for the high-frequency attenuation during audio propagation. Then, a frame-segmentation and windowing operation is performed to divide the continuous audio stream into several audio frames of fixed length. Finally, a Hanning window or Hamming window is added to each audio frame to reduce the spectral leakage problem at the edges of the audio frames. The second step is to extract time-frequency domain features from each preprocessed audio frame, calculate the Mel frequency cepstral coefficients (MFCC) of each frame, filter the spectrum of the audio frame through a Mel scale filter bank, and then perform discrete cosine transform to obtain the cepstral coefficients. The Mel frequency cepstral coefficients of each frame are arranged in chronological order to form a feature sequence of audio data. The Mel frequency cepstral coefficients can accurately capture the time-frequency domain features of the audio signal and are a classic feature in the field of audio recognition. The third step is to input the obtained audio feature sequence into a recurrent neural network (RNN) pre-trained on a large-scale audio event dataset. This network can capture the temporal correlation features of audio features. Through the forward propagation calculation of the network, the network outputs the classification probability distribution of various events corresponding to the current audio data. This probability distribution includes all preset event categories and their corresponding occurrence probabilities. The fourth step is to set probability filtering rules and select the category with the highest probability value in the probability distribution as the environmental event label for the current time period. The preset categories of event labels include typical environmental sound events such as human voices, motor running sounds, collision sounds, and alarm sounds.

[0064] For example, for audio data collected by a robot in an emergency rescue scenario, it is first pre-emphasized and processed by framing and windowing. The Mel frequency cepstral coefficients of each audio frame are calculated and a feature sequence is formed. After the feature sequence is input into a pre-trained recurrent neural network, the network outputs the probability distribution of alarm sound 0.92, collision sound 0.05, and human voice 0.03. The alarm sound with the highest probability value is selected as the current event label.

[0065] Based on the above technical solution, in this embodiment, S1 completes the accurate acquisition and time synchronization of environmental data through multi-source sensing devices, and uses a parallel processing architecture combined with a dedicated algorithm to extract multimodal feature vectors. Event tags are generated through full-process audio signal processing and deep learning classification. The whole process realizes multi-dimensional, high-precision, and real-time perception of the robot's environment, transforming the original environmental sensing data into structured feature vectors and semantic event tags, providing a comprehensive and reliable environmental perception data foundation for subsequent dynamic entity construction, context graph generation, and task planning decisions. At the same time, the algorithm selection and process optimization of each step ensure the efficiency and accuracy of data processing, adapting to the robot's real-time perception needs in dynamic, unstructured, and complex environments.

[0066] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S2 can be implemented through the following S201, S202, and S203, which are explained in detail below: S201. Complete the binding and pairing of multimodal feature vectors with detection results to obtain multiple sets of feature-detection target binding pairs.

[0067] Among them, the binding pair is a dataset formed by associating and binding the target detection results obtained during the robot's environmental perception process with the multimodal feature vectors of the corresponding positions. It is the basic data unit for subsequent spatiotemporal alignment of features and dynamic entity construction. Different types of binding pairs respectively carry the association relationship between two-dimensional visual detection information and three-dimensional point cloud feature information.

[0068] In some implementations, S201 includes the following steps: The first step involves inputting visual data into a target detection neural network pre-trained on a large-scale target detection dataset. Through feature extraction and target localization by the network, two-dimensional object detection bounding boxes for all predefined targets in the image are obtained. These bounding boxes represent the position and extent of the target in the image in coordinate form. Subsequently, each two-dimensional object detection bounding box is bound to the appearance feature vector extracted from the corresponding predefined target position to form a first binding pair. Each first binding pair contains two-dimensional spatial location information and visual appearance feature information. The second step is to use clustering algorithms such as density clustering or hierarchical clustering to cluster the 3D point cloud data collected and preprocessed by S1. By calculating the spatial distance and density distribution between point cloud data, point clouds with similar spatial locations and attributes are divided into the same cluster, resulting in several 3D point cloud clusters. Each cluster corresponds to a physical element in the environment. The third step is to bind each 3D point cloud cluster to the geometric feature vector and structural feature vector extracted from the corresponding location. A 3D point cloud cluster generates a second binding pair and a third binding pair. The second binding pair carries the association between the 3D point cloud spatial cluster and the geometric features, and the third binding pair carries the association between the 3D point cloud spatial cluster and the structural features.

[0069] It should be noted that the construction of binding pairs enables the association of discrete multimodal feature vectors with specific environmental detection targets, giving the originally independent feature vectors a clear spatial orientation and avoiding mismatch between feature vectors and environmental physical elements. At the same time, the construction of binding pairs between the two-dimensional visual detection results and the three-dimensional point cloud clustering results respectively builds a data bridge for subsequent cross-dimensional spatiotemporal alignment and feature correlation determination, which is a key preliminary step for realizing multimodal feature fusion.

[0070] S202. Perform spatial alignment and feature correlation determination on each binding pair, and construct dynamic entities of environmental elements by fusing detection results.

[0071] Among them, spatial alignment is the process of mapping two-dimensional visual detection results and three-dimensional point cloud clustering results to the same spatial coordinate system and determining whether the two point to the same environmental element in physical space; feature correlation determination is the process of verifying whether the feature information of different binding pairs belongs to the same environmental element by calculating the similarity between feature vectors; dynamic entity is a digital model of environmental physical elements that integrates two-dimensional visual information, three-dimensional point cloud information and multimodal feature vectors, which can dynamically represent the features and location attributes of environmental elements.

[0072] In some implementations, S202 includes the following steps: The first step is to perform spatial alignment and feature correlation determination, which includes five sub-steps: Sub-step 1: Calculate the cosine similarity between the appearance feature vector in the first binding pair and the appearance feature vector extracted from the image region corresponding to the 3D point cloud cluster. Use this value as the appearance feature similarity to characterize the degree of matching between the 2D and 3D detection results in visual appearance features. Sub-step 2: Calculate the cosine similarity between the geometric feature vector in the second binding pair and the geometric feature vector extracted from the depth map region corresponding to the 2D object detection bounding box. Use this value as the geometric feature similarity to characterize the degree of matching between the two in spatial geometric features. Sub-step 3: Calculate the cosine similarity between the structural feature vector in the third binding pair and the structural feature vector extracted from the depth map region corresponding to the 2D object detection bounding box. Use this value as the structural feature similarity to characterize the degree of matching between the two in terms of spatial structural features. Sub-step 4: Map the 3D point cloud clusters in the second binding pair to the image plane of the visual data through projection transformation, and calculate the intersection-union ratio of the projection area and the 2D object detection bounding box in the first binding pair. The intersection-union ratio is the intersection area of ​​the projection area and the bounding box divided by the union area. The closer the value is to 1, the higher the spatial location matching degree. Sub-step 5: Set the spatial alignment threshold and feature similarity threshold. If the intersection-union ratio is greater than the preset spatial alignment threshold, and the weighted sum of geometric feature similarity, appearance feature similarity and structural feature similarity is greater than the preset feature similarity threshold, then it is determined that the two-dimensional object detection bounding box and the three-dimensional point cloud cluster originate from the same environmental physical element, forming a matching pair.

[0073] The second step involves constructing dynamic entities based on matching pairs. The spatial features of the 2D object detection bounding boxes in the matching pairs are fused with the 3D point cloud clusters to obtain a fused detection result that includes 2D location and 3D spatial morphology. Then, the fused detection result is comprehensively associated with the appearance, geometric, and structural feature vectors in the corresponding first, second, and third binding pairs. The fused detection result and the multimodal feature vectors are integrated into a whole to jointly construct a dynamic entity representing the physical elements of the environment. Each dynamic entity uniquely corresponds to a physical element in the environment.

[0074] In some implementations, spatial feature fusion is performed between the 2D object detection bounding boxes in the matching pairs and the 3D point cloud clusters, including the following steps: Step 1: Establish a unified spatial coordinate system and complete the coordinate calibration in two-dimensional and three-dimensional space.

[0075] Using the sensor coordinate system of the robot's depth camera / LiDAR as the global 3D world coordinate system, the image pixel coordinate system where the 2D object detection bounding box is located is transformed twice using camera intrinsic and extrinsic calibration parameters to achieve precise alignment with the global 3D world coordinate system. Using the camera intrinsic parameter matrix, the pixel coordinates (u,v) of the 2D bounding box are converted into normalized planar coordinates (x / z,y / z) in the camera coordinate system. Then, combining the depth data (z value) collected by the depth camera / LiDAR and the camera extrinsic parameter matrix, the normalized planar coordinates are further converted into 3D coordinates in the global 3D world coordinate system, so that each vertex and center position of the 2D bounding box has corresponding 3D spatial coordinate information.

[0076] Step 2: Feature mapping and association between 2D bounding boxes and 3D point cloud clusters.

[0077] Based on the calibrated unified coordinate system, the positional features of the 2D object detection bounding box are mapped and associated with the spatial morphological features of the 3D point cloud clusters point by point and region by region, thus clarifying the spatial correspondence between the two: All point cloud data of the 3D point cloud cluster are mapped back to the image pixel coordinate system through projection transformation to obtain its projection area in the 2D image. The projection area is accurately matched with the 2D object detection bounding box in the matching pair, and the point cloud pixel area within the 2D bounding box and the 2D effective area corresponding to the 3D point cloud cluster are marked. Then, the core features of the 2D object detection bounding box (such as the bounding box center coordinates, the coordinates of the four vertices, and the effective range of pixels within the box) and the core features of the 3D point cloud cluster (such as the centroid coordinates of the point cloud cluster, the minimum bounding box, and the coordinates of the contour point cloud) are extracted. The core feature points of the two are then anchored and matched in a unified 3D coordinate system. Next, isolated point clouds and noisy point clouds that are mapped to the outside of the 2D bounding box in the 3D point cloud cluster are removed. At the same time, invalid pixel areas within the 2D bounding box that do not have corresponding 3D point cloud data are removed, and the effective features within the spatially overlapping areas of the two are retained.

[0078] Step 3: Fusion and optimization of two-dimensional and three-dimensional spatial features to generate fusion results.

[0079] The two-dimensional location features, after mapping and invalid feature removal, are complementaryly fused with the three-dimensional morphological features. Feature optimization is then used to obtain the final fused detection result. The fused result simultaneously contains localization information at the two-dimensional pixel level and geometric morphological information at the three-dimensional spatial level. The specific fusion method is as follows: First, the pixel coordinates of the bounding box of the two-dimensional object detection are used as the two-dimensional localization reference to preserve its precise pixel range in the image. The three-dimensional spatial localization reference is the centroid of the three-dimensional point cloud cluster and the three-dimensional coordinates of the minimum bounding box. Combined with the three-dimensional coordinates after the two-dimensional bounding box is transformed, the precise spatial position of the environmental physical elements in the global three-dimensional coordinate system is corrected and determined, forming a two-dimensional + three-dimensional dual localization feature. Then, the three-dimensional point cloud clusters are represented by the point cloud set, spatial contour, and geometric size as core morphological features to fully characterize the three-dimensional spatial morphology of environmental physical elements; the contour features of the two-dimensional object detection bounding box (such as aspect ratio and shape similarity) are used as auxiliary morphological features to supplement and correct the morphological features of the three-dimensional point cloud clusters. For example, if the three-dimensional point cloud has shape loss due to occlusion, the shape prediction of the missing area can be completed by the contour features of the two-dimensional bounding box. Finally, the fused dual localization features and morphological features are structurally encapsulated to form a standardized fusion result, which includes at least: two-dimensional level (bounding box pixel coordinates, effective pixel range, visual contour features), three-dimensional level (point cloud cluster centroid three-dimensional coordinates, minimum bounding box three-dimensional parameters, effective point cloud set, spatial contour features), and global association level (coordinate mapping relationship under a unified three-dimensional coordinate system, feature effective region marking).

[0080] It should be noted that the construction of dynamic entities achieves deep integration of multimodal features and multidimensional detection results, integrating the originally scattered visual, geometric, and structural features with two-dimensional and three-dimensional positional information into a unified digital model, enabling the robot to form a complete and unified cognition of environmental elements.

[0081] S203. Assign multi-dimensional semantic vectors to dynamic entities and generate a contextual graph describing the environment based on the relationships between entities.

[0082] Among them, the multidimensional semantic vector is a feature vector configured for each dynamic entity, containing three types of information: physical attributes, functional attributes, and event labels. It is a semantic representation of the dynamic entity. The context graph is a graph structure model constructed with dynamic entities as nodes and the spatiotemporal and logical relationships between entities as edges. It can describe the characteristics and interrelationships of various elements in the environment in a structured and visual way.

[0083] In some implementations, S203 includes the following steps: The first step is to construct the physical attribute vector of the dynamic entity. Core feature information is extracted directly from the geometric feature vector and structural feature vector associated with the dynamic entity. After feature dimension normalization, the physical attribute vector is formed. This vector fully represents the spatial geometric and structural physical features of the environmental elements corresponding to the dynamic entity. Among them, the core feature information includes the length and direction of the geometric axis of the 3D point cloud, the normal vector of the key plane of the object surface, the distance from the plane to the origin of the coordinate system, the 3D size of the object, spatial contour features and surface structure distribution features. The second step involves constructing the functional attribute vector of the dynamic entity. This involves concatenating the apparent feature vector and physical attribute vector of the dynamic entity, and inputting this concatenated vector into a pre-trained neural network for target recognition and functional attribute prediction. The network outputs the functional attribute vector of the dynamic entity. These functional attributes include graspability score, pushability score, pressability score, rotatability score, standability score, obstacle level, communication cost, and role label. Each score is represented by a value from 0 to 1, indicating the feasibility of the corresponding operation. The remaining attributes are represented by hierarchical or coded forms. For example, a graspability score of 0.85 indicates that the entity has high graspability; obstacle levels are divided into 1-5 levels, with level 1 being unobstructed and level 5 being heavily obstructed; communication cost is represented by a value from 0 to 1, indicating the cost of establishing information interaction with the entity; and role labels use numerical codes to represent the entity's functional role in the environment, such as 001 for the operation target, 002 for the obstacle, and 003 for the auxiliary support. The third step is to construct the event tag vector of the dynamic entity. Based on the event tags obtained by S1, the event tags are digitized by one-hot encoding, which converts the discrete event categories into numerical event tag vectors, thereby realizing the quantitative representation of event information. The fourth step is to configure the multidimensional semantic vector of the dynamic entity. The physical attribute vector, functional attribute vector and event tag vector are concatenated in a preset order to form a unique multidimensional semantic vector for each dynamic entity. This vector realizes the integrated semantic representation of the physical characteristics, functional characteristics and environmental event information of environmental elements. The fifth step involves mining the spatiotemporal relationships between dynamic entities. Each dynamic entity is treated as a graph node, and the node's feature vector is its multidimensional semantic vector. Based on the node's feature vector, the spatial relationship between any two graph nodes is calculated, including distance, azimuth, pitch, and containment relationships. Simultaneously, the temporal relationship between any two graph nodes is obtained based on the chronological order of events in the environment. Specifically, distance is calculated by extracting the 3D center coordinates from the physical attribute vectors of the two dynamic entities and using the Euclidean distance formula to obtain the straight-line spatial distance between the two points. Azimuth is calculated by taking the robot's location as the origin and connecting the 3D center coordinates of the two dynamic entities to the origin. Specifically, this involves performing an arctangent operation on the planar coordinates of the two entities and calculating the difference to obtain their relative azimuth angle on the horizontal plane. Pitch is calculated by performing an arctangent operation on the ratio of the vertical height difference between the 3D center coordinates of the two dynamic entities to the horizontal distance. Containment relationships are determined by judging whether the 3D spatial contour coordinates of one dynamic entity are completely within the 3D spatial contour coordinate range of another dynamic entity, combining the size and spatial position features in the physical attribute vectors for a comprehensive assessment. The sixth step is to construct a context graph. A graph node set is formed by the graph nodes corresponding to all dynamic entities, and an edge set is formed by the labeled edges that represent the spatiotemporal relationships between entities. The labels of the edge set include the node IDs of the two graph nodes that make up the edge, the spatial relationship between the two graph nodes, and the temporal relationship. Then, the graph node set and the labeled edge set together constitute the context graph at the current moment, realizing the structured semantic modeling of the current environment.

[0084] It should be noted that multidimensional semantic vectors endow dynamic entities with rich semantic information, enabling robots not only to perceive the physical form of environmental elements, but also to recognize their functional attributes and the event environment in which they are situated, thus achieving a leap from "perception" to "understanding". Context graphs realize the structured modeling of the environment in a graph structure, explicitly and clearly depicting the spatiotemporal and logical relationships between entities, which can provide a comprehensive and dynamic environmental semantic foundation for subsequent task planning and reasoning, supporting the robot's complex reasoning and decision-making.

[0085] For example, a multi-dimensional semantic vector containing physical attributes, functional attributes, and event labels is configured for the dynamic entities of the table, chair, and water cup. The three are used as graph nodes to calculate spatial relationships such as the distance of 1.2 meters between the table and chair, the azimuth angle of 30°, and the containment relationship between the table and the water cup. Combined with the temporal relationships in the environment without special events, a set of labeled edges is constructed to finally form a contextual graph representing the indoor environment.

[0086] Based on the above technical solution, in this embodiment, S2 achieves preliminary association between multimodal features and detection targets through binding pair construction. A dual verification mechanism of spatial alignment and feature correlation determination achieves accurate matching of multi-dimensional detection results, thereby constructing a dynamic entity that integrates multimodal features. Finally, multi-dimensional semantic vectors are assigned to the dynamic entity, and a contextual graph is generated based on the spatiotemporal relationships between entities. This step achieves deep semantic fusion and structured understanding of the robot's environment, transforming discrete multi-source sensor data into a structured, semantic environmental model. This allows the robot to leap from low-level signal processing to high-level relational cognition, solving the problems of shallow environmental understanding and impoverished and rigid world model information in existing technologies. It lays a solid foundation for subsequent task planning, decision-making, and execution based on environmental cognition.

[0087] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, the above S3 can be implemented through the following S301, S302, and S303, which are explained in detail below: S301. Complete the pre-construction and incremental maintenance of the semantic memory network to form a historical task experience base for the robot.

[0088] Among them, the semantic memory network is a graph structure network built on the incremental experience of the robot's historical task execution. It is used to structurally store task planning schemes, execution processes and results in different situations. It is the basis for the robot to realize experience reuse and analogical reasoning. Its core components are memory units and abstract semantic category nodes.

[0089] In some implementations, the construction of a semantic memory network is divided into two stages: basic construction and incremental maintenance. The specific operations are as follows: (1) Basic construction phase: First, the complete task execution process of the robot is encapsulated into an independent memory unit. Each memory unit contains a context diagram at the start of the task, a hierarchical task tree for execution, a sequence of semantic state changes during task execution, and the final task result, thus realizing the structured storage of single-task experience. Then, the first memory unit is used as the initial node of the network. When there are no abstract semantic category nodes, the network prototype is built directly based on this memory unit.

[0090] (2) Incremental maintenance phase: When the robot completes a new task segment and generates new memory units, it performs the following steps: Step 1: Perform subgraph matching between the initial context graph of the newly added memory unit and the initial context graph of all existing memory units in the semantic memory network, comparing the feature information of the nodes and edges of the two one by one. Step two: For the corresponding node pairs in the subgraph matching process, calculate their semantic similarity using the semantic similarity formula: Where FuncVec is the functional attribute vector, PhyVec is the physical attribute vector, CosineSim is the cosine similarity calculation function used to quantify the similarity between two vectors, and a and b are the indices of the two nodes to be compared. and The preset weighting coefficients satisfy the following conditions: + =1, It is usually set to a larger value to highlight the impact of functional attributes on semantic similarity; Step 3: If the calculated semantic similarity is greater than the preset threshold, the node pair is determined to be semantically similar nodes, and the two are mapped to the same abstract semantic category node. This node is an abstract representation of the same type of semantic entity, realizing the unified classification of similar situations in different memory units. Step four involves connecting the newly added memory units to the corresponding abstract semantic category nodes, while analyzing the relationships between the newly added memory units and the hierarchical task trees in the existing memory units, including inclusion, comparison, causality, etc. Then, based on the relationships, labeled edges are established between the newly added memory units and the existing memory units to complete the incremental expansion of the semantic memory network.

[0091] It should be noted that the semantic memory network adopts an incremental construction method, which does not require retraining on historical data, adapts to the needs of real-time learning of robots, and significantly reduces the computational overhead of network updates; the setting of abstract semantic category nodes realizes the normalization processing of similar situations, allowing robots to reuse experience across tasks, rather than simply replicating the same task; the semantic similarity formula combines functional attributes and physical attributes for judgment, which not only ensures semantic similarity, but also takes into account the matching degree of physical features of entities, making the experience retrieval more accurate.

[0092] For example, if the robot completes two tasks, "grabbing a water cup on the table" and "grabbing a water bottle on the table", and the semantic similarity of the nodes corresponding to the water cup and the water bottle in the initial context graph is 0.93, which is greater than the preset threshold of 0.8, then the two are mapped to the abstract semantic category node of "grabable container". At the same time, the memory units of the two tasks are connected to this node, and an edge of "comparison" relationship is established to complete the incremental maintenance of the network.

[0093] S302. Perform graph similarity matching between the current context graph and the semantic memory network to filter and obtain a set of decision candidate experiences.

[0094] Among them, graph similarity matching is a process of comparing the situational graph of the robot's current environment with the initial situational graph of all memory units in the semantic memory network in a comprehensive manner. The candidate experience set is a set of memory units selected from the semantic memory network whose similarity to the current situational graph is higher than a preset threshold, providing experience reference for the subsequent generation of hierarchical task trees.

[0095] In some implementations, the graph similarity matching and candidate experience set selection are completed in two steps based on the current context graph generated by S2 and the semantic memory network pre-built by S301. The specific operations are as follows: 1. Full graph similarity calculation: Traverse each memory unit in the semantic memory network, match the current context graph with the context graph stored in the memory unit at the beginning time one by one, and calculate the node similarity and edge similarity of the two respectively. The node similarity is calculated based on the semantic similarity formula in S301 to calculate the mean semantic similarity of all corresponding nodes. The edge similarity is calculated based on the label information of the edge (spatial relationship, temporal relationship) using the cosine similarity formula to calculate the mean feature matching degree of the corresponding edge. Then, the node similarity and edge similarity are weighted and summed to obtain the graph similarity score between the two context graphs. 2. Candidate experience set screening: Set a preset threshold for graph similarity, extract and summarize all memory units whose graph similarity scores are greater than the threshold to form a candidate experience set for the current task decision; if the graph similarity scores of all memory units are lower than the preset threshold, the candidate experience set is empty.

[0096] It should be noted that graph similarity matching performs a comprehensive comparison from both node and edge dimensions. Compared with single node matching, it can more accurately represent the overall similarity between two scenarios and avoid mismatches of experience due to similar local features. The combination of full traversal and threshold filtering ensures that no potential similar experience is missed and can quickly filter irrelevant memory units, improving the efficiency of experience retrieval. The selection of candidate experience sets provides a basis for subsequent dual-mode decision-making (experience-driven, rule-based reasoning) and is a key link connecting environmental cognition and task planning.

[0097] For example, if the robot's current task is "to grab a glass from the coffee table", after generating the corresponding current context graph, it performs graph similarity matching with the memory units in the semantic memory network. The graph similarity score of the memory unit "grabbing a water glass on the table" is calculated to be 0.91, and the score of "grabbing a water bottle on the table" is 0.88, both of which are greater than the preset threshold of 0.8. These two memory units are included in the candidate experience set. However, the graph similarity score of the memory unit "removing obstacles on the ground" is 0.35, which is lower than the threshold and is filtered out.

[0098] S303. Based on the matching results of the candidate experience set, reason to generate a hierarchical task tree that is adapted to the current task.

[0099] The hierarchical task tree is a tree structure that decomposes complex tasks into executable parameterized atomic action sequences, with the robot's current task target as the root node. It includes the root node (total task target), sub-target nodes at each level, atomic action nodes, and logical execution relationships between nodes, and serves as the core basis for guiding the robot's action execution. The reasoning process adopts a dual-mode mechanism that combines experience-driven and rule-generated approaches, automatically switching based on whether the candidate experience set is empty.

[0100] In some implementations, based on the candidate experience set obtained from S302, a hierarchical task tree is generated in two cases. A parameter adjustment strategy is designed for the experience-driven mode, and a sub-target sequence generation and atomic action decomposition process is designed for the rule-based generation mode. The specific operations are as follows: Case 1: The candidate experience set is not empty - start the experience-driven mode.

[0101] First, the memory unit with the highest graph similarity in the candidate experience set is taken as the optimal experience template, and the stored context graph is marked as the template context graph, and the stored hierarchical task tree is marked as the template task tree. Then, by comparing the current situational graph with the template situational graph, we can identify the dynamic entities that correspond to each other but have different node feature vectors, and define them as differential dynamic entities. Next, for the atomic actions acting on the differential dynamic entities in the template task tree, parameter adjustments are performed in two categories: First, based on the difference in the physical attribute vectors of the differential dynamic entities, the relevant parameters of the atomic actions are scaled proportionally. For example, when the target entity size increases, the opening and closing degree of the robotic arm gripper is increased proportionally. Second, based on the difference in the spatial relationship vectors of the differential dynamic entities, the target pose parameters of the movement-type atomic actions in the template task tree are transformed by coordinate transformation. Through spatial translation, rotation, and other operations, the transformed parameters are adapted to the actual spatial position of the differential dynamic entities. Finally, the adjusted atomic action parameters are updated to the template task tree, preserving the original task hierarchy and logical execution relationship, and generating a hierarchical task tree adapted to the current task.

[0102] Scenario 2: The candidate experience set is empty - start the rule generation mode.

[0103] First, taking the current task objective as the root node, traverse the current context graph and filter out dynamic entities whose functional attributes satisfy the current task objective, which will be used as the target entities for task execution. Next, combining the functional attribute vector and physical attribute vector of the target entity, a predefined rule base is invoked. The rule base is organized hierarchically by scope and follows the matching priority of spatial relationship rules over object operation rules. First, a preliminary sub-objective is generated according to the spatial relationship rules (such as moving to the target entity's preset range or removing obstacles in the path). Then, the target entity is instantiated into a main operation sub-objective (such as grabbing or pushing the target entity) according to the object operation rules. Subsequently, the main operation sub-objective and the preliminary sub-objective are sorted according to logical dependencies and spatial reachability to form an ordered sequence of sub-objectives. Then, each sub-objective in the sub-objective sequence is decomposed into basic actions that can be executed in the robot motion planning library, and specific execution parameters (such as motion speed, target coordinates, execution time) are configured for each basic action to form a parameterized atomic action sequence. Finally, the current task objective is used as the root node, the ordered sub-objective sequence is used as the intermediate node, and the parameterized atomic action sequence is used as the leaf node. A hierarchical task tree is constructed according to the hierarchical relationship and execution order.

[0104] It should be noted that the experience-driven mode fully utilizes the robot's past successful experiences, which reduces the computational overhead of task planning and avoids the trial-and-error costs of planning from scratch. The parameter adjustment strategy ensures that the experience template is adaptable to the current situation. The rule generation mode enables the robot to autonomously complete task planning even in completely unfamiliar environments. The hierarchical design and priority matching of the predefined rule base ensure the rationality of the sub-goal sequence and the reachability of action execution. The hierarchical task tree structure decomposes complex tasks into simple atomic actions, which conforms to the robot's action execution logic, allowing the robot to complete tasks sequentially and efficiently.

[0105] For example, taking case one: "grabbing a water glass on the table" is the optimal template in the candidate experience set. Compared with the water glass in the template, the glass on the coffee table in the current situation has a larger diameter by 2cm and a spatial coordinate offset of 50cm along the X-axis. The opening and closing degree of the gripper is increased by 2cm proportionally, and the target X coordinate of the movement atomic action is increased by 50cm. After updating, a hierarchical task tree for grasping the glass is generated. Taking case two: When the robot performs the task of "opening the cabinet door" for the first time, the candidate experience set is empty. The scenario graph is traversed to determine "cabinet door" as the target entity. The rule base is called to first generate the preparatory sub-objective "move to 30cm in front of the cabinet door", and then generate the main operation sub-objectives "press the cabinet door handle" and "pull open the cabinet door". After sorting, each sub-objective is decomposed into parameterized atomic actions such as robotic arm movement, finger pressing, and arm stretching. Finally, a hierarchical task tree for opening the cabinet door is constructed.

[0106] Based on the above technical solution, in this embodiment, S3 achieves structured storage and efficient reuse of the robot's historical task experience through incrementally constructed semantic memory network. It accurately filters similar experiences to form a candidate experience set through dual-dimensional graph similarity matching of nodes and edges. Then, through a dual-mode reasoning mechanism combining experience-driven and rule-generation, it generates a hierarchical task tree adapted to the current context. This entire process effectively solves the problems of poor task planning flexibility, low decision-making efficiency, and limited generalization ability in existing technologies. It allows the robot to quickly complete task planning in unfamiliar scenarios using past experience, while also ensuring autonomous reasoning and generation of planning solutions in scenarios with no prior experience. Simultaneously, the parameter adaptive adjustment strategy in the experience-driven mode allows historical experience to flexibly adapt to subtle differences in the current context, while the hierarchical rule base and priority matching in the rule-generation mode ensure the rationality and accessibility of task planning. This implementation improves the robot's task planning efficiency, success rate, and adaptability in dynamic, unstructured, and complex environments, providing core support for the robot's intelligent decision-making and realizing a leap from passively executing pre-programmed instructions to actively making decisions based on experience and rule-based reasoning.

[0107] In one possible implementation of this application embodiment, the above-mentioned S4 specifically includes the following S401 to S403: S401. Execute the parameterized atomic action sequence in the hierarchical task tree, and at the same time preset the semantic state change expectation corresponding to the atomic action.

[0108] Among them, atomic actions are the basic operation units that the robot can directly execute. The parameterized atomic action sequence is an ordered operation sequence with specific execution parameters configured for each atomic action, which is obtained by decomposing the hierarchical task tree generated by S3. The semantic state change expectation is a requirement for semantic attribute changes that are pre-set for each atomic action and are expected to trigger the action entity. It is the core basis for subsequent judgment of the action execution effect.

[0109] In some implementations, S401 includes the following operational steps: Based on the hierarchical task tree generated by S3, the leaf nodes of the tree structure are traversed hierarchically to extract the ordered parameterized atomic action sequence. The robot's actuator executes each atomic action in sequence according to the logical order and time sequence of the action sequence, and maintains real-time data interaction with the environmental perception module during the execution process. At the same time, a corresponding semantic state change set is preset for each atomic action as the semantic state change expectation of the action; the semantic state change set includes three items: first, the target entity, that is, the dynamic entity that the atomic action acts on; second, the attribute index to be changed, that is, the specific attribute dimension in the semantic vector of the dynamic entity that is expected to change; and third, the change expectation value, that is, the specific change value or change trend of the attribute dimension that is expected to occur. Then, the semantic state change expectations of all atomic actions are stored in the execution order to form an expectation sequence corresponding to the atomic action sequence, providing a judgment standard for monitoring the execution effect of subsequent actions.

[0110] It should be noted that the preset semantic state change expectation provides a judgment framework for subsequent semantic level monitoring and action adjustment, enabling the robot to know "what effect the action should achieve", which is a key preliminary step for realizing task-level intelligent feedback.

[0111] For example, the robot executes a hierarchical task tree for "grabbing a glass from the coffee table". The resulting atomic action sequence is "move the robotic arm to 10cm above the glass → adjust the gripper opening to 8cm → close the gripper to grab → raise the robotic arm by 5cm". The expected semantic state change for the atomic action "close the gripper to grab" is: the target entity is the glass, the attribute index to be changed is the spatial position Z-axis coordinate and graspability, and the expected change value is that the Z-axis coordinate increases by ≥3cm and the graspability becomes 1 (complete grasp).

[0112] S402. Collect semantic data during the execution of atomic actions and generate a semantic state change sequence for dynamic entities.

[0113] Among them, the semantic state change sequence is a continuous data sequence in which the semantic attributes of the acting entity change over time during the execution of atomic actions by the robot. It is composed of actual semantic state change fragments corresponding to each atomic action in the execution order, and can truly reflect the actual execution effect of each atomic action and the dynamic changes of the environment.

[0114] In some implementations, based on the semantic state change expectations preset by S401, and combined with the context graph generated by S2 and the real-time perceived environmental data, the semantic state change sequence is generated in four steps, all synchronously with the execution of atomic actions: Step 1: Sample the actual attribute change value. During and after the execution of each atomic action, the semantic vector of the acting entity is sampled in real time at a fixed frequency preset by the robot system. The sampling frequency is adjusted according to the execution duration of the atomic action and the dynamics of the environment. The specific values ​​corresponding to the index of the attribute to be changed in the sampled data are extracted to obtain the actual attribute change value of the acting entity. Step 2: Generate a single-action change fragment. Organize the continuous data of the actual attribute change values ​​over time into a sequence, compare it with the preset change expectation value of the atomic action dimension by dimension, and mark the deviation between the actual change and the expected change to form the actual semantic state change fragment corresponding to the atomic action. This fragment contains core information such as action execution time, actual attribute change value, and deviation value. Step 3: Concatenate to form a complete sequence. Once all atomic actions are completed during the execution of the entire parameterized atomic action sequence, seamlessly concatenate the actual semantic state change fragments corresponding to each atomic action according to the actual execution time sequence of the atomic actions, forming a complete data sequence covering the entire task execution process. Step 4: Dynamically update the context graph. Associate the complete semantic state change sequence with the real-time updated context graph, and synchronously adjust the node feature vectors of the corresponding dynamic entities in the context graph according to the actual changes in semantic state, ensuring that the context graph reflects the actual semantic state of the environment in real time.

[0115] For example, when performing the "close gripper grasp" atomic action, the Z-axis coordinate and graspability of the glass are collected at a sampling frequency of 50ms. Continuous sampling data is obtained, showing the Z-axis coordinate gradually increasing from 0cm to 2.8cm and the graspability increasing from 0.2 to 0.98. After comparing with the preset expected change value, the deviation is marked: the Z-axis coordinate increases by 0.2cm less and the graspability is 0.02 less, forming the actual semantic state change segment of this action. After all grasping-related atomic actions are completed, the change segments of the four actions "move", "adjust gripper", "grasp", and "lift" are spliced ​​together according to time to obtain a complete semantic state change sequence, and the spatial position and attribute information of the glass in the context diagram are updated synchronously.

[0116] S403. Based on the semantic state change sequence and the real-time context graph, dynamically adjust the atomic action parameters and the execution path of the hierarchical task tree.

[0117] This step is the core of the robot's intelligent adaptation during execution. By comparing the actual semantic state change sequence with the preset expectation, the execution effect of the atomic action is determined. For different effects, strategies such as fine-tuning of atomic action parameters or replanning of task execution path are adopted to enable the robot to cope with environmental interference and planning deviations during execution.

[0118] In some implementations, dynamic adjustments are made based on the semantic state change sequence generated by S402, the real-time updated context diagram, and the preset change expectation value of S401, in two categories. Clear adjustment judgment rules and operation procedures are also defined, as follows: Scenario 1: The atomic action does not achieve the expected result but the trend is consistent; the parameters of the executed atomic action are fine-tuned in real time. First, monitor the attribute change values ​​of the entities in the semantic state change sequence in real time. If it is determined that the change trend is completely consistent with the preset change expectation value, but the actual change rate is lower than the system's preset rate threshold, calculate the difference between the actual change value and the change expectation value, and record the difference as the real-time error e(t), where t is the current monitoring time node. Then, a proportional control law is used to fine-tune the parameters of the currently executed atomic actions online. The adjustment formula for the action parameters is as follows: Where u(t) represents the adjusted real-time execution parameters of the atomic action. The baseline execution parameters for this atomic action (preset by the hierarchical task tree). The proportional gain preset for the system is calibrated based on the performance of the robot actuator, the type of motion, and environmental characteristics. Finally, the adjusted parameter u(t) calculated by the formula is sent to the robot's actuator in real time to replace the original baseline parameter. The actuator continues to complete the atomic action according to the new parameter until the attribute change value reaches the preset change expectation value or the action is completed.

[0119] Scenario 2: The execution effect of atomic actions deviates significantly from expectations, requiring replanning of the execution path in the hierarchical task tree. First, a dual failure judgment rule is set. When either rule is detected, the current atomic action is directly judged to have failed: one is that within N consecutive system preset detection cycles, the sign of the real-time error e(t) is always opposite to the sign of the expected change value, that is, the actual change trend of the attribute is completely contrary to the expectation; the other is that the absolute value integral of the real-time error e(t) exceeds the system preset error integral threshold, that is, the cumulative deviation between the actual change of the attribute and the expectation is too large; N is a positive integer, set according to the action execution time. When an atomic action fails to execute, the execution of the current atomic action sequence is immediately paused, the latest updated context graph is retrieved, the graph similarity matching step in S3 is re-executed, the latest context graph is matched with the pre-built semantic memory network, new candidate experience sets are selected and reasoning is performed. Then, based on the results of the rematch, a new hierarchical task tree adapted to the current actual environment is generated according to the dual-mode inference mechanism of S3, and the sub-targets, atomic action sequences, and execution paths of the original task tree are deleted, added, or adjusted. Finally, the original task tree is replaced with the newly generated hierarchical task tree. The robot actuator, based on the atomic action sequence obtained from the decomposition of the new task tree, re-executes the task from the previous logical node of the failed action, thus completing the dynamic adjustment of the task execution path.

[0120] In addition, all adjustment operation records, adjusted parameters, and deviation data of semantic state changes are stored in real time throughout the entire adjustment process, providing data support for the accumulation of subsequent task experience.

[0121] It should be noted that the parameter fine-tuning method of the proportional control law features fast response speed and low computational overhead, adapting to the real-time adjustment needs during robot execution. It can quickly correct small motion deviations and ensure the accuracy of motion execution. Meanwhile, the dual failure judgment rule can accurately identify serious deviations in motion execution, preventing the robot from mechanically executing invalid actions that could lead to task failure or safety risks. The path replanning stage relies on the latest context graph to re-match and reason based on experience, ensuring that the new task tree can adapt to the actual environmental conditions, enabling the robot to cope with sudden environmental changes. The combination of these two adjustment strategies achieves execution control of "fine-tuning for small deviations and replanning for large deviations," realizing intelligent adjustment at the task level.

[0122] Based on the above technical solution, in this embodiment, S4 establishes a basic operation process and effect judgment standard for task execution by sequentially executing parameterized atomic action sequences and pre-setting semantic state change expectations; by sampling semantic data in real time and generating semantic state change sequences, it realizes quantitative monitoring of the execution effect of atomic actions and dynamic updating of environmental semantic states; through parameter fine-tuning of proportional control laws and path replanning based on failure judgment, it realizes two-layer dynamic adjustment from action parameters to task paths during robot execution. The entire process breaks through the limitations of existing technologies where robot execution control lacks high-level semantic monitoring and can only adjust low-level parameters, constructing a closed-loop execution control mechanism that allows the robot to no longer mechanically execute preset actions, but to autonomously optimize execution strategies based on real-time semantic changes in the environment and entities. This implementation effectively improves the compliance, safety, and success rate of robots performing tasks in dynamic, unstructured, and complex environments. At the same time, the real-time stored execution and adjustment data provides rich practical experience for incremental updates of the semantic memory network, helping robots achieve continuous lifelong learning and capability evolution, and providing core execution control support for autonomous, safe, and efficient operation of robots in scenarios such as industrial manufacturing, home services, and emergency rescue.

[0123] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0124] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and modifications.

Claims

1. A robot control method based on artificial intelligence, characterized in that, include: The robot collects visual data, depth data, 3D point cloud data, and audio data of its environment, and processes the data in parallel to obtain multimodal feature vectors and event labels. The multimodal feature vectors are spatiotemporally aligned and associated to construct dynamic entities representing environmental elements. Each dynamic entity is given a multidimensional semantic vector containing physical attributes, functional attributes, and event labels to generate a contextual graph describing the relationships between entities. The context graph and the pre-built semantic memory network are matched for graph similarity. Reasoning is performed based on the matching results to generate a hierarchical task tree. The semantic memory network is built based on the incremental experience of historical task execution and is used to store and associate task planning and results in different contexts. The hierarchical task tree is used to guide the robot's action execution, including parameterized atomic action sequences. Execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence of the dynamic entity and the context graph.

2. The robot control method based on artificial intelligence according to claim 1, characterized in that, The multimodal feature vector includes appearance feature vector, geometric feature vector, and structural feature vector, and the generation process of the multimodal feature vector and the event label includes: The visual data and the depth data are registered, and the registered image is input into a pre-trained convolutional neural network for feature extraction to obtain an appearance feature vector; wherein, the visual data is color image data acquired by an RGB camera; Based on the depth data, the three-dimensional coordinates corresponding to each pixel in the registered image data are calculated, and the length and direction of the geometric axes of the three-dimensional point cloud are extracted by principal component analysis to obtain the geometric feature vector; the depth data is data collected by a depth camera or lidar to characterize the distance from the object surface to the sensor. For the three-dimensional point cloud data, a random sampling consensus algorithm is used to perform plane fitting, and the normal vectors of no less than three maximum planes and the distance from the plane to the origin are extracted to jointly form the structural feature vector; The audio data is first pre-emphasized and framed with windowing. Then, the Mel frequency cepstral coefficients of each frame are calculated to obtain a feature sequence. The feature sequence is input into a pre-trained recurrent neural network, which outputs an event classification probability distribution. The category k with the largest probability value in the probability distribution is taken as the event label for the current time period. The event labels include human voices, motor sounds, collision sounds, and alarm sounds.

3. The robot control method based on artificial intelligence according to claim 2, characterized in that, The step of spatiotemporally aligning and associating the multimodal feature vectors to construct dynamic entities representing environmental elements includes: The visual data is input into a pre-trained target detection neural network to obtain a two-dimensional object detection bounding box of a predefined target, and the bounding box is bound to the appearance feature vector of the corresponding predefined target to obtain a first binding pair; The three-dimensional point cloud data is clustered using a clustering algorithm to obtain three-dimensional point cloud clusters. The three-dimensional point cloud clusters are then bound to the geometric feature vectors and structural feature vectors at corresponding locations to obtain second and third binding pairs. Based on the spatial alignment and feature correlation determination of each binding pair, matching pairs determined to originate from the same environmental physical element are obtained. The two-dimensional object detection bounding boxes in the matching pair are fused with the three-dimensional point cloud clusters, and the fusion result is associated with the multimodal feature vectors in the corresponding first binding pair, second binding pair, and third binding pair to jointly construct a dynamic entity.

4. The robot control method based on artificial intelligence according to claim 3, characterized in that, The spatial alignment and feature correlation determination based on each of the binding pairs includes: The cosine similarity between the appearance feature vector in the first binding pair and the appearance feature vector extracted from the image region corresponding to the point cloud cluster is calculated as the appearance feature similarity. The cosine similarity between the geometric feature vector in the second binding pair and the geometric feature vector extracted from the depth map region corresponding to the bounding box is calculated as the geometric feature similarity. The cosine similarity between the structural feature vector in the third binding pair and the structural feature vector extracted from the depth map region corresponding to the bounding box is calculated as the structural feature similarity. Project the 3D point cloud clusters in the second binding pair onto the image plane, and calculate the intersection-union ratio of the projected region with the 2D object detection bounding boxes in the first binding pair; If the intersection-union ratio is greater than a preset spatial alignment threshold, and the weighted sum of the geometric feature similarity, the appearance feature similarity, and the structural feature similarity is greater than a preset feature similarity threshold, then it is determined that the two-dimensional object detection bounding box and the three-dimensional point cloud cluster originate from the same environmental physical element, forming a matching pair.

5. The robot control method based on artificial intelligence according to claim 1, characterized in that, The step of assigning a multi-dimensional semantic vector containing physical attributes, functional attributes, and event tags to each dynamic entity to generate a contextual graph describing the relationships between entities includes: The physical attribute vector of the dynamic entity is derived from the geometric feature vector and the structural feature vector; The apparent feature vector and physical attribute vector of the dynamic entity are input into a pre-trained neural network for target recognition, and the functional attribute vector of the dynamic entity is output; wherein, the functional attributes include the graspability score, pushability score, pressability score, rotatability score, standing ability score, obstacle level, communication cost, and role label of the dynamic entity. The event tag vector of the dynamic entity is obtained based on the one-hot encoding of the event tag; Each dynamic entity is treated as a graph node, and the node feature vector of the graph node is composed of the physical attribute vector, the functional attribute vector, and the event label vector. The spatial relationship between any two graph nodes is calculated based on the node feature vectors, and the temporal relationship between any two graph nodes is obtained according to the event occurrence time; wherein, the spatial relationship includes distance, azimuth angle, elevation angle, and containment relationship; The current context graph is constructed using a set of graph nodes and a set of labeled edges; wherein the labels of the edge set include the node IDs of the two graph nodes that make up the edge, the spatial relationship between the two graph nodes, and the temporal relationship.

6. The robot control method based on artificial intelligence according to claim 1, characterized in that, The step of performing graph similarity matching between the context graph and the pre-constructed semantic memory network includes: A complete task execution process is encapsulated into a memory unit, which includes a context diagram at the start of the task, a hierarchical task tree, a semantic state change sequence, and the task result. When a new memory unit is added, the initial context graph of the new memory unit is matched with the initial context graph of the existing memory unit. If there are node pairs in the context graphs of the new memory unit and the existing memory unit with a semantic similarity greater than a preset threshold, the node pairs are mapped to the same abstract semantic category node. The newly added memory units are connected to the abstract semantic category nodes. At the same time, based on the association relationship between the newly added memory units and the existing memory units in the hierarchical task tree, edges are established between the newly added memory units and the existing memory units to obtain the semantic memory network. The association relationship includes inclusion, comparison and causality. The current context graph is matched with the context graph at the beginning of each memory unit in the memory network. The graph similarity score is calculated by weighted summation of node similarity and edge similarity between context graphs. Based on a preset similarity threshold, memory units with graph similarity scores greater than the preset similarity threshold are selected as candidate experience sets for the current decision.

7. The robot control method based on artificial intelligence according to claim 6, characterized in that, The step of reasoning based on the matching results to generate a hierarchical task tree includes: If the candidate experience set is not empty, the hierarchical task tree stored in the memory unit with the highest graph similarity is used as the template of the current hierarchical task tree, and the atomic action parameters of the current hierarchical task tree are adjusted according to the node differences between the current context graph and the context graph stored in the memory unit with the highest graph similarity. If the candidate experience set is empty, then with the current task objective as the root node, traverse the entities in the current context graph whose functional attributes satisfy the current task objective, and generate a sub-objective sequence based on the physical attributes and spatial relationships of the entities through a predefined rule base; then decompose each sub-objective into executable parameterized atomic actions in the robot motion planning library to obtain the hierarchical task tree of the current task.

8. The robot control method based on artificial intelligence according to claim 1, characterized in that, The semantic state change sequence is obtained as follows: When executing a hierarchical task tree, a set of semantic state changes is preset for each atomic action, which includes the target entity, the attribute index to be changed, and the expected value of the change. After the atomic action is completed, the semantic vector of the acting entity is sampled at a fixed frequency to obtain the attribute change value of the acting entity; wherein, the acting entity is the entity that the atomic action acts upon. The sequence of recorded attribute change values ​​over time is compared with the expected change value to obtain the actual semantic state change fragment corresponding to the atomic action; By splicing together the actual semantic state change fragments generated by all atomic actions during the entire task execution process in chronological order, a complete semantic state change sequence is obtained.

9. The robot control method based on artificial intelligence according to claim 8, characterized in that, The step of adjusting the action parameters of atomic actions and the task execution path of the hierarchical task tree based on the semantic state change sequence of the dynamic entity and the context graph includes: When the trend of the attribute change value of the active entity is found to be consistent with the expected change value, but the rate of change is lower than the preset rate value, the difference between the two is recorded as the error, and then the action parameter is adjusted using a proportional control law; the adjustment formula for the action parameter u(t) is: ; These are the baseline parameters for atomic actions. For proportional gain, The error is denoted as t, where t represents time. When it is detected that the sign of the error is always opposite to the expected value of the change within N consecutive detection cycles, or the integral of the absolute value of the error exceeds a preset threshold, the atomic action is determined to have failed. Then, graph similarity matching and reasoning are performed again based on the latest context graph to generate a new hierarchical task tree, thereby adjusting the task execution path; where N is a positive integer.

10. A robot control system based on artificial intelligence, characterized in that, include: The module comprises a data acquisition and processing module, a scenario graph construction module, a task planning module, and a task execution module; among which, The data acquisition and processing module is used to acquire visual data, depth data, 3D point cloud data and audio data of the robot's environment, and to process the data of each modality in parallel to obtain multimodal feature vectors and event labels. The context graph construction module is used to perform spatiotemporal alignment and association of the multimodal feature vectors, construct dynamic entities representing environmental elements, and assign a multidimensional semantic vector containing physical attributes, functional attributes and event labels to each dynamic entity to generate a context graph describing the relationships between entities. The task planning module is used to perform graph similarity matching between the context graph and the pre-built semantic memory network, and to perform reasoning based on the matching results to generate a hierarchical task tree; wherein, the semantic memory network is built based on historical task execution experience increments and is used to store and associate task planning and results under different contexts; the hierarchical task tree is used to guide robot action execution, including parameterized atomic action sequences; The task execution module is used to execute the atomic action sequence in the hierarchical task tree, and adjust the action parameters of the atomic actions and the task execution path of the hierarchical task tree according to the semantic state change sequence of the dynamic entity and the context graph.

Images