Intelligent body interaction planning method based on environmental information active perception

Through the agent's embodied interaction planning method based on active perception of environmental information, the interaction characteristic parameters are inferred and expected embodied interaction actions are generated, which solves the instability and security of the agent's action planning in a complex environment, and improves the Adaptability and safety.

CN119077739AActive Publication Date: 2024-12-06TONGJI UNIV

Patent Information

Application Number
CN202411378786.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-12-06
Estimated Expiration
2044-09-30

AI Technical Summary

Technical Problem

The embodied interactive action planning of existing agents in complex, dense, and human settlement environments is difficult to meet the requirements of motion adaptability and safety, especially due to the lack of predictability of interactive action results, resulting in the risk of motion failure and instability.

Method used

Adopt the intelligent interactive planning method based on active perception of environmental information, and by obtaining and aligning environmental information and embodied interaction perception signals, target recognition, semantic segmentation and multimodal information fusion, infer interaction characteristic parameters, and generate expected Embodied interactive actions, using energy gradient methods to generate adaptive actions.

Benefits of technology

It improves the agent's understanding of interactive force-interactive response timing signals, enhances the level of embodied interaction adaptation and the safety of machine actions, and overcomes the problems of unexpected touch and limited visual perception in unknown environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119077739A_ABST
    Figure CN119077739A_ABST
Patent Text Reader

Abstract

The invention relates to an intelligent body interaction planning method based on environment information active perception. The method comprises the following steps: S1, acquiring aligned environment information and information at an interaction interface of an intelligent body and an object; s2, performing multi-modal information fusion representation on the semantic feature information, the agent working environment and the interaction characteristic parameters to obtain environment multi-dimensional information; s3, obtaining a body interaction sensing signal, carrying out predictability judgment on the action of the robot based on the body interaction sensing signal, if the result is within the expectation, generating a series of body interaction actions based on the environment multi-dimensional information, the planned specific task steps and an energy gradient method, and enabling the robot to execute the body interaction actions; on the contrary, a series of interaction actions with interaction force characteristics are generated based on an interaction characteristic parameter energy gradient method, and the robot executes the interaction actions. Compared with the prior art, the method has the advantages of improving the action safety of the machine and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of collaborative intelligent body embodied intelligence, and in particular to an intelligent body embodied interaction planning method based on active perception of environmental information. Background Art

[0002] In recent years, with the significant improvement in the depth and breadth of the application of the intelligent agent industry, its application scenarios have gradually moved from factories to communities and started to work in scenarios where people coexist. There are a large number of interactive work tasks in such work scenarios with unknown, complex, dense, and humanized characteristics. This will put forward higher requirements on the adaptability of interactive actions in the work process of the intelligent agent. Taking the home service intelligent agent as an example, the indoor environment is highly unstructured and dynamic, and the humanized environment is full of various movable objects in the room, such as furniture, shoes, children's toys, etc.; even humans (ourselves) living in these environments and carrying out daily activities pose a major challenge to the autonomous movement of the intelligent agent. In the application process, the intelligent agent must consider not only the level of task completion and motion stability, but also the safety of motion. Ensuring the adaptability and safety of actions in the work scene is a prerequisite for the application of the service intelligent agent. At present, the environmental perception and representation methods based on visual passive single modality often fail to meet the needs of intelligent agent motion planning constraints and embodied interaction interpretability because they ignore the perception of implicit physical properties closely related to embodied interaction. This leads to incoordination or even motion failure in the operation process of the intelligent agent in such scenarios.

[0003] Currently, the main solution for intelligent agent action planning in such scenarios is to store the interactive properties of objects as priors in the latent space of the neural network, use visual observation methods to infer the operational properties of the objects with the trained neural network, and represent them in a semantic form to constrain the embodied interactive actions of the intelligent agent.

[0004] However, the main problem with this type of method is the lack of predictability of the results of the interactive actions, which often leads to unexpected motion failures. It is difficult for the agent to understand the interaction force-interaction response timing characteristics of the object during the embodied interaction process, and thus it is impossible to predict the expected results of the agent's interactive actions, which increases the risk of instability of the agent's interactive actions, and the safety of the machine's interactive actions cannot be fully guaranteed. Summary of the invention

[0005] The purpose of the present invention is to improve the intelligent agent's understanding of the paired interaction force-interaction response timing signals, thereby improving the intelligent agent's embodied interaction adaptation level and improving the safety of machine actions, and to provide an intelligent agent embodied interaction planning method based on active perception of environmental information.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for embodied interaction planning of an intelligent agent based on active perception of environmental information, the method comprising the following steps:

[0008] S1. Acquire environmental information around the intelligent agent controlling the movement of the robot and information at the interface between the intelligent agent and the object, wherein the information at the interface between the intelligent agent and the object is an embodied interaction perception signal, and the environmental information is a two-dimensional image and depth, extract the sampling data at the most recent moment from the acquired information, and align the sampling data to obtain aligned environmental information and information at the interface between the intelligent agent and the object;

[0009] S2, perform target recognition and semantic segmentation on the aligned environmental information to obtain semantic feature information, and fuse the aligned environmental information to obtain the agent working environment, infer the interaction characteristic parameters based on the information at the interaction interface between the aligned agent and the object, perform multimodal information fusion representation on the semantic feature information, the agent working environment and the interaction characteristic parameters, and obtain multi-dimensional information of the environment;

[0010] S3, obtaining an embodied interaction perception signal, and judging the predictability of the robot's action based on the embodied interaction perception signal. If the judgment result is within expectations, a series of first embodied interaction actions are generated based on the multi-dimensional information of the environment and the specific task steps of the plan and an energy gradient method, and the robot performs the first embodied interaction action;

[0011] On the contrary, a series of second interactive actions with interactive force characteristics are generated based on the method of interactive characteristic parameter energy gradient, and the robot performs the second interactive actions.

[0012] Furthermore, the information at the interaction interface between the intelligent agent and the object includes interaction force stimulation during the interaction process, response displacement of the object, temperature and vibration signals, and the embodied interaction perception signal is obtained based on the embodied interaction perception device.

[0013] Furthermore, the spatial position of the embodied interaction sensing device is:

[0014]

[0015] Among them, p i represents the Cartesian space position of the i-th sensor relative to the parent joint, P i represents the position of the i-th sensor relative to the robot's center of mass coordinate system, Represents the homogeneous transformation matrix of the parent joint corresponding to the sensing device relative to the robot's center of mass coordinate system.

[0016] Furthermore, the specific steps of inferring the interaction characteristic parameters based on the information at the interaction interface between the aligned intelligent agent and the object are:

[0017] The information obtained at the interaction interface between the aligned agent and the object is:

[0018]

[0019] Where Δx m is the interaction displacement, is the moving speed of the observation point at the interactive interface, F m is the interaction force at the interface;

[0020] The information obtained above is used to infer the interaction characteristic parameters of the object θ = [K, D, f] T ,During the inference process, first the object interaction characteristics are modeled as:

[0021] Y(t)=H(t)θ+V(t)

[0022] is the system superposition output vector, is the system superposition information matrix, is the system state noise;

[0023] Then, the least squares parameter estimation method estimates the interaction characteristic parameters of the object:

[0024]

[0025] Assume that J1(θ) is When the minimum value is obtained, let J1(θ) The partial derivative of is zero, so we get

[0026]

[0027] Among them, y(j) represents the observed temporal input feature at the interaction interface, v(t) represents the noise, ψ T represents the observed time series input signal at the interactive interface, J1(θ) represents the system prediction error, K represents the object elastic operation coefficient, D represents the object damping operation coefficient, and f represents the object operation loss constant.

[0028] Furthermore, the specific steps of judging the predictability of the robot action based on the embodied interaction perception signal are:

[0029] The planner first generates the target tracking point g* at the next moment based on the target position g:

[0030]

[0031] Among them, ε(x) is the effective solution space of the planner in Cartesian space, and x is the current position of the robot;

[0032] Determine whether the robot's movement speed and the interaction force with the outside world obtained based on the embodied interaction perception signal exceed the maximum safe speed v when the robot approaches the target point safe and the maximum interaction force F safe , the steps to calculate the interaction force are: calculate the acceleration of the robot when tracking the target point

[0033]

[0034] Where M and C represent the inertia matrix and Coriolis matrix respectively, q represents the robot joint space velocity, and G represents the gravity vector;

[0035] So the speed of the robot when tracking the target point is Interaction

[0036] The predictability of the robot's action is judged, and the judgment result of the robot's action a is obtained.

[0037] Furthermore, the judgment result of the robot action a is:

[0038]

[0039] where a d It is a dangerous action, which means it is not expected. s For safe action, the expression is within expectations.

[0040] Furthermore, the system prediction error is:

[0041]

[0042] Among them, y(j) represents the observed temporal input feature at the interaction interface, v(t) represents the noise, ψ T represents the observed timing input signal at the interactive interface, and J1(θ) represents the system prediction error.

[0043] Furthermore, the energy gradient-based method generates a series of first embodied interaction actions, and the robot performs the first embodied interaction actions specifically as follows:

[0044] First, the robot's workspace is topologically transformed into the robot body domain Obstacle area and free motion space domain in represents the robot's position, r and h represent the robot's spatial configuration at the current position, and x represents the robot's current position. represents the robot's workspace, represents the obstacle domain, represents the machine ontology domain, represents the difference between the workspace and the obstacle domain and the robot body domain;

[0045] Subsequently, different spatial topologies are assigned different energy states, where the target domain is assigned a virtual energy potential field with elastic gravity. The free motion space domain is defined as having a linear damping field U f (x), the obstacle domain is defined as U with energy operation cost o,d (x) = W(θ, x) work domain,

[0046] By deriving the energy state, the driving instruction of the robot is obtained as:

[0047] F*=-grad[U(x)]

[0048] The driving instruction of the robot is a first embodied interactive action.

[0049] Furthermore, the energy state is the energy state of the robot when the spatial position is x, specifically:

[0050]

[0051] Among them, U o (x) represents U o,d (x) is a collection of

[0052] Furthermore, the method based on the energy gradient of the interaction characteristic parameter generates a series of second interaction actions with interaction force characteristics, specifically:

[0053] Calculate the second interaction action specific F act , the specific characteristics of the second interaction are determined by the sine function:

[0054] F act =F safe sin(ωt), t∈(0, 100]

[0055] Where w = π / 100 is the frequency of the interaction change, and t is the number of sampling times.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] By actively applying a series of interactive force stimuli to the object, the intelligent agent can observe the interactive force-interaction response timing characteristics of the object after the stress through the multimodal sensing device. Based on these observation characteristics, the intelligent agent can infer the implicit interactive characteristics of the object and then constrain the interactive actions of the intelligent agent, improve the intelligent agent's understanding of the paired interactive force-interaction response timing signals, and then improve the intelligent agent's embodied interactive adaptation level and improve the safety of machine actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A diagram of the multimodal perception system of the present invention;

[0059] Figure 2 This is a multimodal perception flow chart of the present invention;

[0060] Figure 3 is the environmental information reasoning flow chart of the present invention;

[0061] Figure 4 Generate a flow chart for the embodied interaction planning action of the present invention;

[0062] In the figure, there are a multimodal perception module 110, a perception communication and command issuance module 120, an environmental information reasoning module 130, an embodied interaction planning module 140, and a perception-motion coordination module 150. DETAILED DESCRIPTION

[0063] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0064] The present invention proposes an intelligent agent embodied interaction planning method based on active perception of environmental information, the method comprising the following steps:

[0065] S1. Acquire environmental information around the intelligent agent controlling the movement of the robot and information at the interface between the intelligent agent and the object, wherein the information at the interface between the intelligent agent and the object is an embodied interaction perception signal, and the environmental information is a two-dimensional image and depth, extract the sampling data at the most recent moment from the acquired information, and align the sampling data to obtain aligned environmental information and information at the interface between the intelligent agent and the object;

[0066] S2, perform target recognition and semantic segmentation on the aligned environmental information to obtain semantic feature information, and fuse the aligned environmental information to obtain the agent working environment, infer the interaction characteristic parameters based on the information at the interaction interface between the aligned agent and the object, perform multimodal information fusion representation on the semantic feature information, the agent working environment and the interaction characteristic parameters, and obtain multi-dimensional information of the environment;

[0067] S3, obtaining an embodied interaction perception signal, and judging the predictability of the robot's action based on the embodied interaction perception signal. If the result is within expectations, a series of embodied interaction actions are generated based on the energy gradient method based on the multi-dimensional information of the environment and the specific task steps of the plan, and the robot performs the embodied interaction actions;

[0068] On the contrary, a series of interactive actions with interactive force characteristics are generated based on the method of energy gradient of interactive characteristic parameters, and the robot performs the interactive actions.

[0069] The present invention enables the intelligent agent to have the ability to perceive physical signals at the interactive interface like a human. By applying active interactive force stimulation to the object, the intelligent agent can observe the spatial state reaction of the object after stress through control means such as electronic skin vision. The intelligent agent's understanding of the paired interactive force-interactive response timing signal can infer the implicit physical properties of the object. By actively acquiring information on the interactive characteristics of the surrounding environment, the robot has the ability to predict interactive actions. Based on this ability, the intelligent agent will overcome the intelligent agent perception-motion control problems such as accidental touch, limited visual perception, and unsolvable free movement space brought about by unknown, complex, dense, and human-inhabited working scenes.

[0070] The above method can be implemented by the following multimodal perception system, such as Figure 1 As shown, the system includes:

[0071] The multimodal perception module 110 is used to effectively collect the local working environment information of the intelligent agent to meet the needs of the embodied intelligent agent's environmental perception and the intelligent agent's motion planning constraints;

[0072] The sensing communication and command issuing module 120 is responsible for the information collection and transmission of the distributed multi-mode sensing devices and the sending of the intelligent motion control intelligence;

[0073] The environmental information reasoning module 130 processes the environmental information collected by the multimodal perception module to realize the embodied intelligent body environment modeling and the inference of the environmental interaction characteristics;

[0074] The embodied interaction planning module 140 generates an embodied interaction motion sequence based on the agent's task and environmental perception reasoning constraints to realize active perception and finally complete the interaction task;

[0075] The perception-motion coordination module 150 determines the predictability of the current intelligent agent's embodied interaction actions, coordinates and activates the active interaction perception and task operation planning units corresponding to the embodied interaction planning.

[0076] Specifically, the multimodal perception module 110 can perceive the environment and the multi-dimensional information of the interaction between the intelligent agent and the environment, and then transmit the perceived data to the environmental information reasoning module 130 and the perception-motion coordination module 150 through the perception communication and instruction issuance module 120 after time-space consistency calibration to meet the needs of the intelligent agent's embodied action generation.

[0077] The perception communication and command issuing module 120 is responsible for the data path of the whole body of the intelligent body, transmits the multimodal perception module 110 to the unit that needs to subscribe, and sends the action command generated by the embodied interaction planning module 140 to the joint module to complete the embodied action.

[0078] The environmental information reasoning module 130 collects and fuses the collected environmental information to build a map, and provides the embodied interaction planning module 140 with environmental constraints for action generation, thereby ensuring the predictability of the intelligent body's actions.

[0079] The embodied interaction planning module 140 drives the robot to generate embodied interaction actions according to the task planning requirements, the environmental information constraints integrated by the environmental information reasoning module 130, and the action generation unit activated by the perception-motion coordination module 150.

[0080] The perception-motion coordination module 150 evaluates the predictability of the agent's interactive actions based on the information features at the interactive interface collected by the multimodal perception module 110. Based on the evaluation results, the corresponding action generation unit of the embodied interaction planning module 140 is activated to generate action instructions.

[0081] In one embodiment of the present invention: the multimodal perception module 110 includes: an RGB-D visual perception unit 1102: used to perform geometric and image perception of the surrounding environment of the embodied intelligent body to meet the needs of large-scale rapid spatial recognition and mapping; an embodied interaction perception unit 1103: used to collect object signals at the interaction interface, observe the spatial state reaction of the object after stress, and complete the pairing of interaction force-interaction response timing signals; a spatiotemporal consistency alignment unit 1104: after the data collected by the multi-type perception devices are aligned through timing, the spatiotemporal consistency calibration is completed, and the unified observation signal is transmitted to the environmental information reasoning module 120 through the perception communication and instruction release module.

[0082] Specifically, the specific implementation of the multimodal perception module 110 ( Figure 2 )as follows:

[0083] RGB-D visual perception unit 1102 collects the environment information around the agent in the form of two-dimensional images and depth, and then prints the collected visual data with a timestamp and saves it in the corresponding queue;

[0084] The embodied interaction perception unit 1103 collects information at the interaction interface between the agent and the object, including: interaction force stimulation during the interaction process, response displacement of the object, temperature, vibration and other signals for time series collection, and then prints the collected multi-dimensional data with a timestamp and stores it in the form of a dictionary into the corresponding queue;

[0085] The spatiotemporal consistency alignment unit 1104 extracts the most recent sampling data from the data storage queue of the RGB-D visual perception unit 1102 and the data storage queue of the embodied interaction perception unit 1103 at a specific sampling frequency, aligns the time dimension through the reference sensor alignment method, and sends the aligned data to the perception communication and instruction issuing module 120.

[0086] Considering the spatial consistency problem of the distributed perception of the embodied interaction perception unit 1103, based on the forward kinematic chain method, the spatial position of each embodied interaction perception device is calculated by obtaining the joint position encoding information of the robot:

[0087]

[0088] In one embodiment of the present invention: the environmental information reasoning module 130 ( Figure 3 ), including: a semantic reasoning unit 1302: reasoning about the semantic information of the environment to identify the semantic attributes of objects in the environment; an environmental geometry reconstruction unit 1303: modeling the information of the environment around the agent based on the geometric information and appearance information of multimodal environmental perception to achieve reconstruction of a three-dimensional geometric map; an interactive characteristic reasoning unit 1304: understanding the interactive force-interaction response timing signal collected by the body interactive perception to complete the inference of implicit characteristics of objects that are closely related to the embodied interactive attributes that are difficult to observe visually; and a fusion mapping unit 1305: fusing the constructed three-dimensional map, the semantic map and the inferred object interaction features to construct a multi-dimensional environmental information map.

[0089] Specifically, the specific implementation of the environmental information reasoning module 130 is as follows:

[0090] Semantic reasoning unit 1302: Based on the convolutional neural network technology, it performs target recognition and semantic segmentation with the collected images, and then integrates the segmented semantic features with the object model reconstructed by the environment geometry reconstruction unit 1303 to realize the semantic feature reasoning of the object.

[0091] Environmental geometry reconstruction unit 1303: Based on three-dimensional reconstruction technology (Nerf neural network, Gaussian sputtering surface, etc.), the collected RGB image and depth point cloud information are integrated to perceive the geometric information of the intelligent agent environment and complete the geometric reconstruction of the intelligent agent working environment.

[0092] Interaction characteristic reasoning unit 1304: Based on the interaction force-interaction response time series characteristics collected by the embodied interaction perception unit 1103, a data-driven system identification method is used to complete the implicit feature parameter inference of objects closely related to the interaction characteristics.

[0093] Fusion mapping unit 1305: Based on the semantic feature information extracted by semantic reasoning unit 1302, the environmental geometry information reconstructed by environmental geometry reconstruction unit 1303, and the interaction characteristic parameters inferred by interaction characteristic reasoning unit 1304, a multimodal information fusion representation method is used to perform environmental information and fusion mapping. The formed multi-dimensional environmental information will be used for action generation of embodied interaction planning module 140.

[0094] In one embodiment of the present invention: the embodied interaction planning module 104 includes: an embodied interaction action generation unit 1402: generating a series of embodied interaction instructions to control the intelligent body to apply a series of interaction actions to the interacted object, and complete the interaction force-interaction response time series feature collection; a task operation action generation unit 1403: generating a series of intelligent body operation actions with predictable interaction properties based on environmental information constraints and the task instructions of the intelligent body.

[0095] Specifically, the specific implementation of the embodied interaction planning module 140 is as follows ( Figure 4 ):

[0096] In order to ensure the coordination and consistency between the embodied interactive action generation unit 1402 and the task operation action generation unit 1403, the perception-motion coordination module 150 first senses the data according to the multimodal perception module 110, and uses the judgment result to activate the corresponding embodied interactive action generation unit. When the action of the intelligent body is within the expected range, the task operation action generation unit 1403 will be activated, otherwise, the embodied interactive action generation unit 1402 will be activated.

[0097] The embodied interaction action generation unit 1402: infers the object interaction characteristic parameters facing the interaction characteristic reasoning unit 1304, and generates a series of interaction actions with interactive force characteristics based on the energy gradient method, so as to guide the intelligent body to apply force stimulation to the object, and then the embodied interaction perception unit 1103 completes the perception of the object's interaction characteristic information.

[0098] Task operation action generation unit 1403: Aiming at the execution of intelligent agent tasks in complex scenarios, based on the multimodal information map generated by the environmental information reasoning module 130 and the specific task steps planned, a series of embodied interactive actions are generated based on the energy gradient method, and the problem of intelligent agent movement stability caused by unknown, complex, dense, and human-inhabited operation scenarios is overcome by transforming the spatial state of the intelligent agent itself and the surrounding environment.

[0099] The specific steps of inferring the interaction characteristic parameters based on the information at the interaction interface between the aligned intelligent agent and the object are:

[0100] The information obtained at the interaction interface between the aligned agent and the object is:

[0101]

[0102] Where Δx m is the interaction displacement, is the moving speed of the observation point at the interactive interface, F m is the interaction force at the interface;

[0103] The information obtained above is used to infer the interaction characteristic parameters of the object θ = [K, D, f] T ,During the inference process, it is assumed that the projection of the observed data to the parameter vector is a linear model, and their parameters can be determined by linear fitting methods. The object interaction characteristics are modeled as:

[0104] Y(t)=H(t)θ+V(t)

[0105] is the system superposition output vector, is the system superposition information matrix, is the system state noise.

[0106] Then, the least squares parameter estimation method estimates the interaction characteristic parameters of the object:

[0107]

[0108] Assume that J1(θ) is When the minimum value is obtained, let J1(θ) The partial derivative of is zero, so we get

[0109]

[0110] When the action is in an unexpected range,

[0111] The energy gradient-based method generates a series of embodied interactive actions as follows:

[0112] The purpose of active tactile sensing is to apply a series of force stimuli to obstacles and observe their special shapes to obtain sufficient perception data sets. The input of active sensing is the interaction force F act , and its interaction force variation characteristics are determined by the sine function:

[0113] F act =F safe sin(ωt), t∈(0, 100]

[0114] Where w = π / 100 is the frequency of the interaction change, and t is the number of sampling times.

[0115] When the action is within the expected range,

[0116] A series of interactive actions with interactive force characteristics are generated based on the method of interactive characteristic parameter energy gradient:

[0117] First, the robot's workspace is topologically transformed into the robot body domain Obstacle area and free motion space domain in represents the robot's position, r and h represent the robot's spatial configuration at the current position, and x represents the robot's current position. represents the robot's workspace, represents the obstacle domain, represents the machine ontology domain, represents the difference between the workspace and the obstacle domain and the robot body domain;

[0118] Subsequently, different spatial topologies are assigned different energy states, where the target domain is assigned a virtual energy potential field with elastic gravity. The free motion space domain is defined as having a linear damping field U f (x), the obstacle domain is defined as U with energy operation cost o,d (x) = W(θ, x) work domain,

[0119] So the energy state of the robot when its spatial position is x is:

[0120]

[0121] Among them, U o (x) represents U o,d (x) a collection of;

[0122] By deriving the energy state, the driving instruction of the robot is obtained as:

[0123] F * = -grad[U(x)]

[0124] The driving instruction of the robot is an interactive action with interactive force characteristics.

[0125] The specific steps of inferring the interaction characteristic parameters based on the information at the interaction interface between the aligned intelligent agent and the object are:

[0126] The information obtained at the interaction interface between the aligned agent and the object is:

[0127]

[0128] Where Δx m is the interaction displacement, is the moving speed of the observation point at the interactive interface, Fm is the interaction force at the interface;

[0129] The information obtained above is used to infer the interaction characteristic parameters of the object θ = [K, D, f] T ,During the inference process, first the object interaction characteristics are modeled as:

[0130] Y(t)=H(t)θ+V(t)

[0131] is the system superposition output vector, is the system superposition information matrix, is the system state noise;

[0132] Then, the least squares parameter estimation method estimates the interaction characteristic parameters of the object:

[0133]

[0134] Assume that J1(θ) is When the minimum value is obtained, let J1(θ) The partial derivative of is zero, so we get

[0135]

[0136] Among them, y(j) represents the observed temporal input feature at the interaction interface, v(t) represents the noise, ψ T represents the observed time series input signal at the interactive interface, J1(θ) represents the system prediction error, K represents the object elastic operation coefficient, D represents the object damping operation coefficient, and f represents the object operation loss constant.

[0137] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for embodied interaction planning of an intelligent agent based on active perception of environmental information, characterized in that: The method comprises the following steps: S1. Obtaining environmental information around an intelligent agent controlling the movement of the robot and information at an interface between the intelligent agent and an object, wherein the information at the interface between the intelligent agent and the object is an embodied interaction perception signal, and the environmental information is a two-dimensional image and depth, extracting the sampling data at the most recent moment from the acquired information, and aligning the sampling data to obtain aligned environmental information and information at an interface between the intelligent agent and the object; S2, perform target recognition and semantic segmentation on the aligned environmental information to obtain semantic feature information, and fuse the aligned environmental information to obtain the agent working environment, infer the interaction characteristic parameters based on the information at the interaction interface between the aligned agent and the object, perform multimodal information fusion representation on the semantic feature information, the agent working environment and the interaction characteristic parameters, and obtain multi-dimensional information of the environment; S3, obtaining an embodied interaction perception signal, and judging the predictability of the robot's action based on the embodied interaction perception signal. If the judgment result is within expectations, a series of first embodied interaction actions are generated based on the multi-dimensional information of the environment and the specific task steps of the plan and an energy gradient method, and the robot performs the first embodied interaction action; On the contrary, a series of second interactive actions with interactive force characteristics are generated based on the method of interactive characteristic parameter energy gradient, and the robot performs the second interactive actions.

2. According to claim 1, a method for embodied interaction planning based on active perception of environmental information, characterized in that: The information at the interaction interface between the intelligent agent and the object includes the interaction force stimulation of the interaction process, the response displacement of the object, temperature and vibration signals, and the embodied interaction perception signal is obtained based on the embodied interaction perception device.

3. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 2, characterized in that: The spatial position of the embodied interactive sensing device is: Among them, p i represents the Cartesian space position of the i-th sensor relative to the parent joint, P i represents the position of the i-th sensor relative to the robot's center of mass coordinate system, Represents the homogeneous transformation matrix of the parent joint corresponding to the sensing device relative to the robot's center of mass coordinate system.

4. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 1, characterized in that: The specific steps of inferring the interaction characteristic parameters based on the information at the interaction interface between the aligned intelligent agent and the object are: The information obtained at the interaction interface between the aligned agent and the object is: Where Δx m is the interaction displacement, is the moving speed of the observation point at the interactive interface, F m is the interaction force at the interface; The information obtained above is used to infer the interaction characteristic parameters of the object θ = [K, D, f] T ,During the inference process, first the object interaction characteristics are modeled as: Y(t)=H(t)θ+V(t) is the system superposition output vector, is the system superposition information matrix, is the system state noise; Then, the least squares parameter estimation method estimates the interaction characteristic parameters of the object and obtains the system prediction error J1(θ), Assume that J1(θ) is When the minimum value is obtained, let J1(θ) The partial derivative of is zero, so we get Among them, y(j) represents the observed temporal input feature at the interaction interface, v(t) represents the noise, ψ T represents the observed time series input signal at the interactive interface, J1(θ) represents the system prediction error, K represents the object elastic operation coefficient, D represents the object damping operation coefficient, and f represents the object operation loss constant.

5. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 1, characterized in that: The specific steps of judging the predictability of the robot action based on the embodied interaction perception signal are: The planner first generates the target tracking point g* at the next moment based on the target position g: Among them, ε(x) is the effective solution space of the planner in Cartesian space, and x is the current position of the robot; Determine whether the robot's movement speed and the interaction force with the outside world obtained based on the embodied interaction perception signal exceed the maximum safe speed v when the robot approaches the target point safe and the maximum interaction force F safe , the steps to calculate the interaction force are: calculate the acceleration of the robot when tracking the target point Where M and C represent the inertia matrix and Coriolis matrix respectively, q represents the robot joint space velocity, and G represents the gravity vector; So the speed of the robot when tracking the target point is Interaction The predictability of the robot's action is judged, and the judgment result of the robot's action a is obtained.

6. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 5, characterized in that: The judgment result of the robot action a is: where a d It is a dangerous action, which means it is not expected. s For safe action, the expression is within expectations.

7. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 6, characterized in that: The system prediction error is: Among them, y(j) represents the observed temporal input feature at the interaction interface, v(t) represents the noise, ψ T represents the observed timing input signal at the interactive interface, and J1(θ) represents the system prediction error.

8. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 1, characterized in that: The energy gradient-based method generates a series of first embodied interactive actions, and the robot performs the first embodied interactive actions specifically as follows: First, the robot's workspace is topologically transformed into the robot body domain Obstacle area and free motion space domain in represents the robot's position, r and h represent the robot's spatial configuration at the current position, and x represents the robot's current position. represents the robot's workspace, represents the obstacle domain, represents the machine ontology domain, represents the difference between the workspace and the obstacle domain and the robot body domain; Subsequently, different spatial topologies are assigned different energy states, where the target domain is assigned a virtual energy potential field with elastic gravity. The free motion space domain is defined as having a linear damping field U f (x), the obstacle domain is defined as U with energy operation cost o,d (x) = W(θ, x) work domain, By deriving the energy state, the driving instruction of the robot is obtained as: F*=grad[U(x)] The driving instruction of the robot is a first embodied interactive action.

9. The method for planning intelligent agent embodied interaction based on active perception of environmental information according to claim 8, characterized in that: The energy state is the energy state of the robot when the spatial position is x, specifically: Among them, U o (x) represents U o,d (x) is a collection of 10. The method for embodied interaction planning of an intelligent agent based on active perception of environmental information according to claim 1, characterized in that: The method based on the energy gradient of the interaction characteristic parameter generates a series of second interaction actions with interaction force characteristics as follows: Calculate the second interaction action specific F act , the specific characteristics of the second interaction are determined by the sine function: F act =F safe sin(ωt),t∈(0,100] Where w = π / 100 is the frequency of the interaction change, and t is the number of sampling times.

Citation Information

Patent Citations

  • Movement assistance device

    CN104582668A

  • Operation skill learning method based on agent active interaction

    CN116719409A

  • Force feedback interaction method for energy consistency of multiple virtual coupling systems

    CN118484089A

  • Systems and methods for interacting with a visual cell

    US20150269855A1

Cited By

  • Heterogeneous historic building intelligent real-time monitoring and early warning device based on multiple sensors

    CN119984404A

  • Interaction method and system of space intelligent flying robot in microgravity environment

    CN120593770A

  • Robot control reconstruction method and system based on PLC replacement

    CN121625172A