Interactive object recognition method, system and equipment and medium

Abstract

The invention relates to an interactive object recognition method, system and device and a medium. The method comprises the steps that S1, body sensing data of an actuator installed at the tail end of a robot is read in real time; s2, preprocessing the collected body sensing data based on a physical model to recover spatial features of the body sensing data, and encoding a preprocessing result; s3, analyzing the encoded body sensing data by using an object recognition model constructed based on a transformer model to obtain a hidden space representation representing an interactive object category; in the training process of an object recognition model constructed based on a transformer model, body sensing data is enhanced through a data enhancement module based on knowledge and physical joint constraint, so that embedding of a basic physical rule is realized; and S4, decoding the hidden space representation representing the category of the interactive object, and identifying the interactive object. Compared with the prior art, the method improves the accurate recognition of interactive objects in visual occlusion and narrow space.

Description

Technical Field

[0001] This invention relates to the field of object recognition technology, and in particular to an interactive object recognition method, system, device and medium. Background Technology

[0002] Currently, many tasks involving the state estimation or recognition of interactive objects mainly rely on RGB cameras, RGB-D cameras, or other environmental sensors based on optical principles.

[0003] A search revealed that Chinese invention patent application CN110142590A discloses a method for predicting the next interactive object based on a transformer first-person perspective. However, this method has a single perceptual modality, and all its capabilities are built on a single visual signal.

[0004] However, in real-world applications, robots often face complex situations such as drastic changes in lighting, complete darkness, strong reflections, visual occlusion, and confined spaces. Under these adverse conditions, traditional visual sensors, limited by their imaging mechanisms, struggle to acquire sufficiently high-quality environmental information, leading to a significant decrease in object recognition accuracy or even complete failure. Especially in dark scenes or confined environments with severe occlusion, robots lose task continuity due to their inability to reliably perceive the state of interactive objects, severely limiting their reliable application in all weather conditions and scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing an interactive object recognition method, system, device and medium. This method further improves the accuracy of interactive object recognition in visual occlusion and confined spaces, and has higher robustness.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, an interactive object recognition method is provided, comprising:

[0008] S1. Real-time reading of body perception data from the actuator installed at the robot's end effector;

[0009] S2. Perform physical model-based preprocessing on the collected body perception data to recover the spatial features of the body perception data, and encode the preprocessing results;

[0010] S3. Use the object recognition model built on the transformer model to parse the encoded body perception data to obtain the latent space representation of the interactive object category; during the training process of the object recognition model built on the transformer model, the body perception data is enhanced by the data augmentation module based on knowledge and physical constraints, thereby realizing the embedding of basic physical laws.

[0011] S4. Decode the latent space representation of the interactive object category to identify the interactive object.

[0012] Preferably, the type of body perception data is determined by the sensors mounted on the actuator.

[0013] Preferably, the types of body perception data include proprioceptive perception data and tactile perception data.

[0014] Preferably, the step of preprocessing the acquired body perception data based on a physical model to recover the spatial features of the body perception data specifically includes:

[0015] Construct the 3D state of the actuator, align the collected body perception data with the actuator's spatial state, and align the tactile sensor readings to the specific locations in the 3D model. If the body perception data categories cannot be accurately aligned, no processing is performed.

[0016] Preferably, during the training process of the object recognition model built based on the transformer model, the body perception data is enhanced by a data augmentation module based on knowledge and physical constraints, thereby achieving the embedding of basic physical laws.

[0017] The data augmentation module based on knowledge and physical constraints includes a body perception data generator based on a generative model and a verifier based on a simulation engine. The generator is used to generate body perception data that conforms to the laws of human cognition, and the verifier is used to ensure that the generated data conforms to the physical constraints of the real world.

[0018] Preferably, the body perception data generator based on the generative model uses a diffusion model or a VAE model as the backbone network. For generating long sequence data or spatial structure data, the diffusion model is used as the backbone network, and in other cases, the VAE model is used as the backbone network.

[0019] According to a second aspect of the present invention, an interactive object recognition and pose estimation system based on humanoid body perception is provided, employing the above-described method, the system comprising:

[0020] The data acquisition module is used to read body perception data from the actuators installed at the robot's end effector in real time.

[0021] The encoding module is used to perform physical model-based preprocessing on the acquired body perception data to recover the spatial features of the body perception data, and to encode the preprocessing results.

[0022] The parsing module is used to parse the encoded body perception data using an object recognition model built on the transformer model, and obtain the latent space representation of the interactive object category.

[0023] The decoding module decodes and identifies interactive objects from the latent space representation of their states.

[0024] Preferably, during the training process of the object recognition model built based on the transformer model in the parsing module, the body perception data is enhanced by the data augmentation module based on knowledge and physical constraints, thereby realizing the embedding of basic physical laws.

[0025] The data augmentation module based on knowledge and physical constraints includes a body perception data generator based on a generative model and a verifier based on a simulation engine. The generator is used to generate body perception data that conforms to the laws of human cognition, and the verifier is used to ensure that the generated data conforms to the physical constraints of the real world.

[0026] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement any of the methods described above.

[0027] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described herein.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) By using the body perception data of the actuator installed at the end of the robot, an interactive object perception method that does not rely on vision is adopted, which avoids the problem that the robot has difficulty recognizing interactive objects in adverse lighting, visual obstruction and narrow space.

[0030] (2) An object recognition model based on the transformer model is used to parse the encoded body perception data to obtain the latent space representation of the interactive object category, and then the interactive object information is identified by encoding. This solves the problem that the traditional recognition model has insufficient understanding of complex body perception data and cannot estimate the interactive object information.

[0031] (3) A data augmentation module based on knowledge and physical constraints was introduced during the training of the object recognition model, which solved the problem of poor availability of body perception data and ensured that the generated data met both human operating habits and the physical constraints of the real world. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] Example

[0035] like Figure 1 As shown in this embodiment, an interactive object recognition method is provided, which includes:

[0036] S1. Read body perception data from the actuator installed at the robot's end effector in real time.

[0037] In this embodiment, the type of body perception data is determined by the sensors mounted on the actuator, including body perception data and tactile perception data.

[0038] This embodiment can integrate various body perception sensors (such as six-dimensional force / torque sensors, inertial measurement units (IMUs), tactile arrays, etc.) to acquire multi-dimensional body perception data such as force, acceleration, posture, and contact area during the interaction between the robot and the object.

[0039] Specifically, proprioceptive data represents the perception of one's own state, such as the motion posture, joint state, and actuation status of a robot's end effector (dexterous hand). The core is to provide feedback on "what posture the hand is in now and how much it has moved," similar to a human's perception of the position of their fingers and the angle of their joints (proprioception). See Table 1 below for details.

[0040] Table 1

[0041]

[0042] Specifically, tactile perception data focuses on information about the contact between the dexterous hand and external objects and the environment, as shown in Table 2 below.

[0043] Table 2

[0044]

[0045]

[0046] S2. Perform physical model-based preprocessing on the collected body perception data and encode the processing results.

[0047] The purpose of physical model-based preprocessing is to recover the spatial features of body perception data. Specifically, this involves: constructing the 3D state of the actuator and aligning the collected body perception data with the spatial state of the actuator, such as aligning the readings of the tactile sensor to the specific location of the 3D model and aligning the state of the body perception to the pose of the 3D model; if certain categories of body perception data cannot be accurately aligned, they will not be processed, such as temperature information collected using only a single temperature sensor.

[0048] By constructing a physical field-based position encoding, the position information of an object is mapped to its physical properties and spatial constraints, thereby enabling a more accurate object position-physical feature association to be established in the Transformer architecture.

[0049] In this embodiment, a variational autoencoder (VAE) is used to encode the processing results.

[0050] S3. Use the object recognition model built based on the transformer model to parse the encoded body perception data to obtain the latent space representation of the interactive object category.

[0051] In this embodiment, the object recognition based on the transformer model adopts a hybrid perception Transformer model based on a multi-head self-attention mechanism. Cross-modal feature interaction and semantic fusion are achieved through a cross-attention module. The attention mechanism is used to solve the sparsity of the captured ontology perception and tactile perception data, and the encoded body perception data is parsed.

[0052] By leveraging the multimodal cross-attention mechanism of the decoder in Transformer, key interaction features in body perception data are dynamically extracted, thereby improving the robust recognition capability of interactive objects in complex environments.

[0053] In model training, it is necessary to first embed basic physical laws using a large-scale dataset so that the model can grasp the basic principles of object estimation using body perception. The purpose of pre-training is to reduce oscillations during the training process and improve the stability of the output results.

[0054] Due to the limited availability of body perception data, it is necessary to enhance this data. This embodiment proposes a data augmentation module based on joint knowledge and physical constraints. Its structure includes a body perception data generator based on a generative model and a verifier based on a simulation engine. The generator aims to produce body perception data that conforms to human cognitive patterns, while the verifier aims to ensure that the generated data conforms to real-world physical constraints.

[0055] Specifically, the body perception data generator based on the generative model uses a diffusion model or a variational autoencoder (VAE) model as the backbone network. When it is necessary to generate long sequence data or spatial structure data, the diffusion model is used as the backbone network, and in other cases, a lightweight VAE model is used as the backbone network.

[0056] Furthermore, during the training process, occlusion scene data is generated by introducing a physical simulation environment, and a reward function based on physical rules is designed to enable the model to better combine physical constraints for inference when facing occlusion.

[0057] S4. Decode the latent space representation of the interactive object category to identify the interactive object.

[0058] In this embodiment, a multilayer perceptron (MLP) is used to decode the latent space representation of the interactive object category, thereby identifying the interactive object.

[0059] To verify the performance of the method of this invention, this embodiment was tested on various dexterous hands, including but not limited to dexterous hands with different degrees of freedom, different sensor types, and different actuation methods, such as the shadow hand, the Inspirerobot hand, and the OYMotion hand. In the experiments, this example tested various experimental scenarios, including but not limited to daily life scenarios and factory production scenarios. This embodiment achieved an accuracy of 92.2% in the object recognition task, not only achieving results similar to visual algorithms in estimation accuracy but also avoiding the problems of illumination influence and occlusion. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. The present invention also includes technical solutions composed of any combination of the above technical features.

[0060] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0061] Multiple components in the device are connected to the I / O interface, including: input units such as keyboards and mice; output units such as various types of displays and speakers; storage units such as disks and optical discs; and communication units such as network interface cards (NICs), modems, and wireless transceivers. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0062] The processing unit executes the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 by any other suitable means (e.g., by means of firmware).

[0063] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0064] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0065] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0066] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An interactive object recognition method, characterized in that, include: S1. Real-time reading of body perception data from the actuator installed at the robot's end effector; S2. Perform physical model-based preprocessing on the collected body perception data to recover the spatial features of the body perception data, and encode the preprocessing results; S3. Use an object recognition model built on the transformer model to parse the encoded body perception data to obtain the latent space representation of the interactive object category; wherein, during the training process of the object recognition model built on the transformer model, the body perception data is enhanced by a data augmentation module based on knowledge and physical constraints, thereby realizing the embedding of basic physical laws. S4. Decode the latent space representation of the interactive object category to identify the interactive object.

2. The interactive object recognition method according to claim 1, characterized in that, The type of body perception data is determined by the sensors mounted on the actuator.

3. The interactive object recognition method according to claim 2, characterized in that, The types of body perception data include proprioceptive perception data and tactile perception data.

4. The interactive object recognition method according to claim 1, characterized in that, The preprocessing of the acquired body perception data based on a physical model to recover the spatial features of the body perception data specifically includes: Construct the 3D state of the actuator, align the collected body perception data with the actuator's spatial state, and align the tactile sensor readings to the specific locations in the 3D model. If the body perception data categories cannot be accurately aligned, no processing is performed.

5. The interactive object recognition method according to claim 1, characterized in that, During the training process of the object recognition model built on the transformer model, the body perception data is enhanced by a data augmentation module based on knowledge and physical constraints, thereby achieving the embedding of basic physical laws. The data augmentation module based on knowledge and physical constraints includes a body perception data generator based on a generative model and a verifier based on a simulation engine. The generator is used to generate body perception data that conforms to the laws of human cognition, and the verifier is used to ensure that the generated data conforms to the physical constraints of the real world.

6. The interactive object recognition method according to claim 5, characterized in that, The body perception data generator based on the generative model uses a diffusion model or a VAE model as the backbone network. For generating long sequence data or spatial structure data, the diffusion model is used as the backbone network, while in other cases, the VAE model is used as the backbone network.

7. An interactive object recognition system, employing the method described in claim 1, characterized in that, The system includes: The data acquisition module is used to read body perception data from the actuators installed at the robot's end effector in real time. The encoding module is used to perform physical model-based preprocessing on the acquired body perception data to recover the spatial features of the body perception data, and to encode the preprocessing results. The parsing module is used to parse the encoded body perception data using an object recognition model built on the transformer model, and obtain the latent space representation of the interactive object category. The decoding module decodes and identifies interactive objects from the latent space representation of their states.

8. The system according to claim 7, characterized in that, The data augmentation module based on knowledge and physical constraints includes a body perception data generator based on a generative model and a verifier based on a simulation engine. The generator is used to generate body perception data that conforms to the laws of human cognition, and the verifier is used to ensure that the generated data conforms to the physical constraints of the real world.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images