Unified characterization and extraction method for multi-modal feature information

Abstract

The invention relates to a multi-modal feature information unified characterization and extraction method, which comprises the following steps of S1, monitoring whether a current intelligent agent of a robot is in physical contact with an external object or not, and if the current intelligent agent is in contact with the external object and an executed task is an interactive task, generating an implicit feature inference instruction; s2, performing time sequence pairing on the interaction force data characteristics and the interaction displacement-interaction speed characteristics to obtain interaction force-relative speed-interaction displacement time sequence data at an interaction interface; and S3, constructing an implicit information representation model, fitting representation parameters of the model by adopting a gradient descent method, and obtaining an object space grid feature prediction value based on the model when an error result is smaller than the confidence degree measurement matrix. Compared with the prior art, the method has the advantages that the robustness and generalization ability of intelligent agent control of the robot are improved, and the like.

Description

Technical Field

[0001] The present invention relates to the field of collaborative intelligent body embodied intelligence, and in particular to a method for unified representation and extraction of multimodal feature information. Background Art

[0002] In recent years, with the increasing demand and cases of intelligent agents in unstructured environments, unstructured scenes, with their cluttered space, many types of objects, and high safety requirements, pose a major challenge to the effective operation of intelligent agents. One of the main reasons for the failure of intelligent agent movement is that the intelligent agent physically interacts with unknown objects in the environment during operation, which causes the movement of the intelligent agent to deviate from the pre-planned results. Sudden stops in movement and damage to objects often occur. In order to cope with the huge challenges posed by this complex environment to movement, the intelligent agent needs to fully consider the physical process of interaction between itself and objects in the environment. The implicit characteristics hidden in the object are difficult to directly observe through visual methods, but they significantly affect the spatial state of the intelligent agent and the object itself during the physical interaction. How to incorporate this information into the perception range of the intelligent agent and introduce it into the motion planning method of the intelligent agent in an appropriate form is the key to solving this problem.

[0003] Currently, the main solutions to this problem include: implicitly constructing a direct mapping from visual observations to agent actions such as RT1, RT2, etc. through supervised learning methods; and constructing a closed loop from behavior to reward through unsupervised methods such as reinforcement learning, gradually forming a behavioral strategy based on tasks and observations.

[0004] However, due to the unclear intrinsic mapping mechanism of neural networks and insufficient interpretability, it is still difficult to evaluate and measure whether these implicit characteristics closely related to physical interactions are fully captured in the perception process and used to constrain the motion generation of robots. The generalization ability and operability of such methods still need to be tested in further applications. Summary of the invention

[0005] The purpose of the present invention is to provide a unified representation and extraction method of multimodal feature information in order to realize online perception of implicit features of objects and improve the robustness and generalization ability of robot intelligent body control.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for uniformly representing and extracting multimodal feature information, the method comprising the following steps:

[0008] S1. Monitor whether the current intelligent body of the robot has physical contact with external objects. If so, and the task being performed is an embodied interaction task, generate implicit feature inference instructions;

[0009] S2, the visual perception device, tactile perception device, and proprioceptive perception device of the intelligent body obtain multimodal perception data based on implicit feature inference instructions, wherein the multimodal perception data includes tactile interaction force observation signals, proprioceptive signals, visual observation signals, and tactile point cloud signals, and the multimodal perception data are aligned, and the interaction force data features at the interaction interface are extracted from the aligned tactile interaction force observation signals;

[0010] Extracting the interactive displacement-interaction velocity features at the interactive interface from the registered proprioceptive signals;

[0011] Reconstruct the registered visual observation signal and tactile point cloud signal to obtain the object space grid feature;

[0012] The interactive force data features and the interactive displacement-interactive velocity features are paired in time series to obtain the interactive force-relative velocity-interactive displacement time series data at the interactive interface;

[0013] S3. Initialize the object feature parameter matrix and construct an implicit information representation model. The input of the model is the interaction force-relative velocity-interaction displacement time series data at the interaction interface, and the output is the object space grid feature at the next moment. The gradient descent method is used to fit the model's representation parameters. When the error result is less than the confidence measurement matrix, the object space grid feature prediction value is obtained based on the model.

[0014] Furthermore, in the registration process, the signals of the two devices are registered as follows:

[0015]

[0016] Among them, x aibj ——The signal after aligning the sampling time of sensor a to the reference sensor b; x ai ——the sampling value of sensor a at time i; T a ——Sampling period of sensor a; t bj ——The sampling time of sensor b at time j; t ai ——The sampling time of sensor a at time i.

[0017] Furthermore, the object space grid feature is the spatial form of an object in grid form, the number of vertices of the grid of the object space grid feature is n, and each vertex has a corresponding object implicit feature parameter matrix.

[0018] Furthermore, the interactive displacement-interaction speed characteristics at the interactive interface obtained by extracting the registered proprioceptive signal are specifically:

[0019]

[0020] Where Δx m is the interaction displacement, is the moving speed of the observation point on the interactive interface.

[0021] Furthermore, the interaction force data features at the interaction interface obtained by extracting the registered tactile interaction force observation signal are specifically as follows:

[0022]

[0023] Where Δx i is the displacement of the jth vertex of the object mesh during the interaction process, is the velocity of the jth vertex of the object mesh during the interaction, F i is the force received by the jth vertex of the object mesh during the interaction process.

[0024] Furthermore, the gradient descent method is used to fit the characterization parameters of the model. When the error result is less than the confidence measurement matrix, the predicted value of the object space grid feature is obtained based on the model:

[0025] The implicit feature parameters of an object are characterized based on the mesh vertices of the object space mesh features. The implicit feature parameter matrix of each vertex is Therefore, the implicit physical system model of the object based on mesh vertex representation is modeled as:

[0026] Y(t)=H(θ)X(t)+V(t)

[0027] in, is the state output of the mesh vertex, is the state input of the mesh vertices, is the system state parameter matrix, is the system state noise;

[0028] Calculate the system modeling error, and update the system state parameter matrix by taking the partial derivative of the system modeling error. When the system modeling error E(θ) is less than the confidence measure matrix E(θ) min Stop parameter updating.

[0029] Furthermore, the system modeling error is:

[0030]

[0031] in, is the state output of the mesh vertex, is the state input of the mesh vertices, is the system state parameter matrix, and n represents the number of mesh vertices.

[0032] Furthermore, the partial derivative of the system modeling error is:

[0033]

[0034] Where θ is the implicit feature parameter matrix.

[0035] Furthermore, the system state parameter matrix is ​​updated as:

[0036]

[0037] Where E(θ) represents the system modeling error.

[0038] Furthermore, the method also includes: subscribing to the robot's current high-level task execution primitives in real time through the ROS topic subscription mechanism, and when the job type is an embodied interactive job, starting to monitor whether the robot's current intelligent body has physical contact with external objects.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] Compared with the existing direct end-to-end mapping methods from perception to instruction, the present invention perceives and represents the implicit characteristics of objects together with the apparent geometric characteristics, giving the intelligent agent the ability to measure characteristics that are closely related to the physical interaction characteristics. Based on this ability, the intelligent agent will have a more robust ability to evaluate the stability of embodied interactions, effectively overcoming the problem of unclear and insufficient interpretability of the inherent mapping mechanism of neural networks. In addition, compared with the existing object feature extraction methods based on visual perception, the present invention is based on multimodal perception signals obtained by visual, tactile and proprioceptive sensors, and has the ability to reveal the inherent physical laws that affect the changes in the spatial state of objects. Combined with the data-driven system feature inference method, the intelligent agent has the ability to perceive the implicit characteristics of objects online, and by adaptively constructing parameter evidence for feature points and fitting the feature parameters, the intelligent agent has the ability to predict the spatial state of objects in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A diagram of a feature information extraction system of the present invention;

[0042] Figure 2 It is the embodied interactive perception flow chart of the present invention;

[0043] Figure 3 The interactive event triggering flow chart of the present invention;

[0044] Figure 4 This is a flowchart of implicit information extraction of the present invention. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] The present invention proposes a unified representation and extraction method of multimodal feature information based on implicit characteristics of scene events (such as Figure 1 As shown), this method enables the intelligent agent to have the ability to perceive and extract implicit features of the object. Through the observed multimodal perception data (vision, touch, proprioception), the intelligent agent is able to reconstruct the spatial form of the interactive object. Subsequently, based on the characterization of the mesh vertices, the implicit characteristics of the object are characterized as a feature parameter matrix with a specific size. Based on the paired interaction force-relative velocity-interaction displacement perception feature information and the corresponding reconstructed object mesh, the intelligent agent infers and fits the characteristic parameters of the object. This method can effectively alleviate the problem of unclear and insufficient interpretability of the intrinsic mechanism of the end-to-end mapping of perception signals and behavior actions, and improve the robustness of the embodied interaction actions of the intelligent agent in unstructured environments. The specific steps of the present invention include:

[0047] S1. Monitor whether the current intelligent body of the robot has physical contact with external objects. If so, and the task being performed is an embodied interaction task, generate implicit feature inference instructions;

[0048] S2, the visual perception device, tactile perception device, and proprioceptive perception device of the intelligent body obtain multimodal perception data based on implicit feature inference instructions, wherein the multimodal perception data includes tactile interaction force observation signals, proprioceptive signals, visual observation signals, and tactile point cloud signals, and the multimodal perception data are aligned, and the interaction force data features at the interaction interface are extracted from the aligned tactile interaction force observation signals;

[0049] Extracting the interactive displacement-interaction velocity features at the interactive interface from the registered proprioceptive signals;

[0050] Reconstruct the registered visual observation signal and tactile point cloud signal to obtain the object space grid feature;

[0051] The interactive force data features and the interactive displacement-interactive velocity features are paired in time series to obtain the interactive force-relative velocity-interactive displacement time series data at the interactive interface;

[0052] S3. Initialize the object feature parameter matrix and construct an implicit information representation model. The input of the model is the interaction force-relative velocity-interaction displacement time series data at the interaction interface, and the output is the object space grid feature at the next moment. The gradient descent method is used to fit the model's representation parameters. When the error result is less than the confidence measurement matrix, the object space grid feature prediction value is obtained based on the model.

[0053] The above steps can be implemented by the following system, which includes:

[0054] Implicit information representation model 110: Based on the object grid representation method, the parameter matrix method is used to describe the implicit characteristics of the object and enhance the robustness of the intelligent agent in the physical interaction process;

[0055] Interaction event trigger module 120: monitors the current tasks performed by the intelligent agent and the observation values ​​of the main body sensors in real time, and activates the implicit inference system for objects when a physical interaction event is detected and an interaction operation is required;

[0056] Embodied interactive perception module 130: obtains the time series results of multi-dimensional sensor observations and completes the raw sensor data processing for the spatiotemporal consistency calibration of multi-modal sensors;

[0057] Implicit information extraction module 140: completes implicit feature parameter matrix inference, through which the multimodal perception data input is used to predict the spatial state of the object;

[0058] Specifically, the implicit information representation model 110 represents the spatial form of the object in the form of a grid, where the number of vertices of the grid is n, and the implicit feature parameter description matrix for each vertex is R 3×3 . Therefore, the implicit feature network parameter matrix of the object is P∈R n×3×3 .

[0059] The interactive event trigger module 120 subscribes to the current agent's high-level task primitives in real time. When the robot's current task is whether it is an embodied interactive task, it also monitors in real time whether the current agent is in physical contact with external objects. When the agent is in contact with the outside world and the task it is performing is an embodied interactive task, the interactive event trigger module sends an implicit feature inference instruction to the embodied interactive perception module 130 to activate the embodied interactive perception module.

[0060] After receiving the embodied interaction perception instruction from the interaction event triggering module 120, the embodied interaction perception module 130 obtains the paired object space state grid data and the interaction force-relative velocity-interaction displacement feature data at the interaction interface by processing the multimodal perception data, and sends these data to the implicit information extraction module 140 for the object implicit feature inference.

[0061] The implicit information extraction module 140, after receiving the interactive force-relative velocity-interactive displacement features and the object space state grid data processed by the embodied interaction perception module 130, initializes the object implicit feature parameter matrix. Then, the implicit feature network parameter matrix is ​​P∈R n×3×3 Perform feature inference and complete implicit feature perception of objects.

[0062] In one embodiment of the present invention: the interactive event trigger module 120 includes multiple sub-modules, including: a job task monitoring module 1202: used to subscribe to the current intelligent agent's high-level job task primitives; an interactive event monitoring module 1203: monitors whether the current intelligent agent has physical contact with external objects.

[0063] In one embodiment of the present invention: the embodied interaction perception module 130 includes multiple sub-modules, including: a spatiotemporal consistency calibration module 1302: after the observed interaction interface data is aligned through time sequence, the spatiotemporal consistency calibration is completed; an interaction force extraction module 1303: feature extraction of interaction force features during the interaction process; an interaction motion extraction module 1304: extracting interaction displacement-interaction speed characteristics at the interaction interface based on the proprioception module; an object space state reconstruction module 1305: based on visual and touch perception data, the spatial state of the object is reconstructed and represented as a grid format; a perception feature pairing module 1306: completing the time sequence pairing of interaction force-relative speed-interaction displacement at the interaction interface.

[0064] In one embodiment of the present invention: the implicit information extraction module 140 includes multiple sub-modules, including: a parameter matrix self-organization module 1402: determining the size of the parameter matrix according to the object representation grid vertices reconstructed by the embodied interaction perception module; a parameter inference and fitting module 1403: inferring and fitting the parameters in the parameter matrix according to the time sequence of the integrated interaction force-relative velocity-interaction displacement and the reconstructed object grid.

[0065] Specifically, the specific implementation of the interactive event triggering module 120 ( Figure 3 )as follows:

[0066] The job task monitoring module 1202 subscribes to the robot's current high-level task execution primitive in real time through the ROS topic subscription mechanism. When the job type is an embodied interactive job, the subscription result is returned as True and sent to the interactive event monitoring module 1203;

[0067] The interactive event monitoring module 1203 receives tactile and proprioceptive signals to monitor in real time whether there is any object interaction between the intelligent body and the external environment. When the intelligent body has physical interaction and the task monitoring module 1202 returns a result of True, the embodied interaction perception module 130 is activated to start the implicit feature inference of the object.

[0068] Specifically, the specific implementation of the embodied interaction perception module 130 ( Figure 2 )as follows:

[0069] The spatiotemporal consistency calibration module 1302 receives data observed by the visual perception device, tactile perception device, and proprioceptive device of the intelligent agent, and saves these data into a queue after encapsulating the timestamp, and then performs spatiotemporal registration on the multimodal perception data. The registration steps are as follows:

[0070]

[0071] Where: x aibj ——Align the sampling time of sensor a to the reference sensor b;

[0072] x ai ——The sampling value of sensor a at time i;

[0073] T a ——Sampling period of sensor a;

[0074] t bj ——The sampling time of sensor b at time j;

[0075] t ai ——The sampling time of sensor a at time i;

[0076] The spatial consistency alignment method is:

[0077]

[0078] Interaction force extraction module 1303: receives the tactile interaction force observation signal aligned by the spatiotemporal consistency calibration module 1302, extracts the interaction force data features at the interaction interface based on the tactile time trigger mechanism; and completes the spatial encoding of the tactile perception signal based on the forward kinematic chain rule;

[0079] Interaction motion extraction module 1304: receives the proprioceptive signal aligned by the spatiotemporal consistency calibration module 1302, and extracts the interaction displacement-interaction velocity features at the interaction interface based on the forward kinematic chain and Jacobian matrix method;

[0080] Object space state reconstruction module 1305: receives the visual observation signal and the tactile point cloud signal aligned by the spatiotemporal consistency calibration module 1302, and uses them for object space state reconstruction;

[0081] The perception feature pairing module 1306 receives the interaction force feature, object space grid feature, interaction displacement-interaction speed feature after data feature extraction, and completes the time sequence pairing of interaction force-relative speed-interaction displacement at the interaction interface.

[0082] After the collected signal is sent to the time-space consistency calibration module 1302, time-space consistency alignment is performed including spatial coordinate alignment and time coding alignment.

[0083] Specifically, the specific implementation of the implicit information extraction module 140 ( Figure 4 )as follows:

[0084] The parameter matrix self-organizing module 1402 receives the object space state representation grid data sent by the embodied interaction perception module 130 and detects whether the implicit model parameters of the object are initialized. If it is detected that the object is not initialized, the object feature parameter matrix is ​​initialized. The size of the feature parameter matrix is ​​P∈R n×3×3 , where n is the number of vertices used to represent the object mesh;

[0085] The parameter inference fitting module 1403 receives the object space state representation grid and the time series data of the interaction force-relative velocity-interaction displacement at the interaction interface sent by the embodied interaction perception module 130. The module uses the time series data of the interaction force-relative velocity-interaction displacement as the model input and the object space state representation grid at the next moment as the output, and uses the gradient descent method to fit the representation parameters of the model. When the error result is less than the confidence measurement matrix, the object implicit feature parameter perception is completed.

[0086] In the present invention, the specific steps of reconstructing the registered visual observation signal and tactile point cloud signal to obtain the object space grid feature are:

[0087] After obtaining the calibrated observation images, image retrieval is first performed using k-means and approximate nearest neighbor search methods, and the most similar images in the image library are obtained by fast (hundreds of image pairs per second) spatial verification and then estimating the approximate affine transformation between the query image and the result image. Subsequently, the initial query results are obtained using DAAT (single document) to cluster in scale space. A method based on detail retrieval and Structure-from-Motion (SfM) system is combined to perform spatial verification through affine transformation estimation to obtain a quantitative measure of scene overlap. Next, geometric verification of image pairs, that is, the test of pairwise geometry in SfM, operates in projective space to estimate the necessary or fundamental matrix of the moving camera and the homography matrix of the purely static camera. The initial 3D model is obtained by explicitly mining the original reconstruction. Next, the image pixels are back-projected into 3D points into the world coordinate system to complete the vision-based object mesh reconstruction. Subsequently, a tactile touch method is used for the low-resolution part of the 3D model to obtain the touch point cloud and align it with the visual point cloud to obtain the visual-tactile fused object point cloud. Finally, the object point cloud is reconstructed based on the Ball-Pivoting Algorithm (BPA) method.

[0088] The gradient descent method is used to fit the representation parameters of the model. When the error result is less than the confidence measurement matrix, the predicted value of the object space grid feature is obtained based on the model:

[0089] The implicit physical features of the object are characterized based on the mesh vertices, and the implicit feature parameters of each vertex are Therefore, the implicit physical system model of the object based on mesh vertex representation is modeled as:

[0090] Y(t)=H(θ)X(t)+V(t)

[0091] in, is the state output of the mesh vertex, is the state input of the mesh vertices, is the system state parameter matrix, is the system state noise.

[0092] So the system modeling error is:

[0093]

[0094] By taking partial derivatives of the state parameter matrix, we get:

[0095]

[0096] The steps for updating parameters are:

[0097]

[0098] When the parameter error E(θ) is less than E(θ) min Stop parameter updating.

[0099] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for unified representation and extraction of multimodal feature information, characterized in that: The method comprises the following steps: S1. Monitor whether the current intelligent body of the robot has physical contact with external objects. If so, and the task being performed is an embodied interaction task, generate implicit feature inference instructions; S2, the visual perception device, tactile perception device, and proprioceptive perception device of the intelligent body obtain multimodal perception data based on implicit feature inference instructions, wherein the multimodal perception data includes tactile interaction force observation signals, proprioceptive signals, visual observation signals, and tactile point cloud signals, and the multimodal perception data are aligned, and the interaction force data features at the interaction interface are extracted from the aligned tactile interaction force observation signals; Extracting the interactive displacement-interaction velocity features at the interactive interface from the registered proprioceptive signals; Reconstruct the registered visual observation signal and tactile point cloud signal to obtain the object space grid feature; The interactive force data features and the interactive displacement-interactive velocity features are paired in time series to obtain the interactive force-relative velocity-interactive displacement time series data at the interactive interface; S3. Initialize the object feature parameter matrix and construct an implicit information representation model. The input of the model is the interaction force-relative velocity-interaction displacement time series data at the interaction interface, and the output is the object space grid feature at the next moment. The gradient descent method is used to fit the model's representation parameters. When the error result is less than the confidence measurement matrix, the object space grid feature prediction value is obtained based on the model.

2. A method for unified representation and extraction of multimodal feature information according to claim 1, characterized in that: In the registration process, the signals of the two devices are registered as follows: Among them, x aibj ——The signal after aligning the sampling time of sensor a to the reference sensor b; x ai ——the sampling value of sensor a at time i; T a ——Sampling period of sensor a; t bj ——The sampling time of sensor b at time j; t ai ——The sampling time of sensor a at time i.

3. The method for unified characterization and extraction of multimodal feature information according to claim 1, characterized in that: The object space grid feature is the spatial form of an object in grid form. The number of vertices of the grid of the object space grid feature is n, and each vertex has a corresponding object implicit feature parameter matrix.

4. The method for unified characterization and extraction of multimodal feature information according to claim 1, characterized in that: The interactive displacement-interaction speed characteristics at the interactive interface obtained by extracting the registered proprioceptive signal are specifically: Where Δx m is the interaction displacement, is the moving speed of the observation point on the interactive interface.

5. A method for unified representation and extraction of multimodal feature information according to claim 4, characterized in that: The interaction force data features at the interaction interface obtained by extracting the registered tactile interaction force observation signal are as follows: Where Δx i is the displacement of the jth vertex of the object mesh during the interaction process, is the velocity of the jth vertex of the object mesh during the interaction process, F i is the force received by the jth vertex of the object mesh during the interaction process.

6. A method for unified representation and extraction of multimodal feature information according to claim 3, characterized in that: The gradient descent method is used to fit the representation parameters of the model. When the error result is less than the confidence measurement matrix, the predicted value of the object space grid feature is obtained based on the model: The implicit feature parameters of an object are characterized based on the mesh vertices of the object space mesh features. The implicit feature parameter matrix of each vertex is Therefore, the implicit physical system model of the object based on mesh vertex representation is modeled as: Y(t)=H(θ)X(t)+V(t) in, is the state output of the mesh vertex, is the state input of the mesh vertices, is the system state parameter matrix, is the system state noise; Calculate the system modeling error, and update the system state parameter matrix by taking the partial derivative of the system modeling error. When the system modeling error E(θ) is less than the confidence measure matrix E(θ) min Stop parameter updating.

7. A method for unified representation and extraction of multimodal feature information according to claim 6, characterized in that: The system modeling error is: in, is the state output of the mesh vertex, is the state input of the mesh vertices, is the system state parameter matrix, and n represents the number of mesh vertices.

8. A method for unified representation and extraction of multimodal feature information according to claim 7, characterized in that: The partial derivative of the system modeling error is: Where θ is the implicit feature parameter matrix.

9. A method for unified representation and extraction of multimodal feature information according to claim 8, characterized in that: The system state parameter matrix is ​​updated as: Where E(θ) represents the system modeling error.

10. The method for unified representation and extraction of multimodal feature information according to claim 1, characterized in that: The method also includes: subscribing to the robot's current high-level task execution primitives in real time through the ROS topic subscription mechanism, and when the job type is an embodied interactive job, starting to monitor whether the robot's current intelligent body has physical contact with external objects.

Images