Intelligent agent multi-dimensional ontology incremental cognitive system and method

Abstract

The invention relates to an agent multi-dimensional ontology incremental cognitive system and method. The system comprises an ontology motion sensing module, an ontology cognition evaluation module, a memory pool module, an ontology knowledge mining module and an ontology cognition cultivation module. Wherein the body motion sensing module is used for obtaining multi-modal data; the ontology cognition evaluation module is used for obtaining an ontology cognition quantization error by using the multi-modal data; the memory pool module is used for data storage; the ontology knowledge mining module is used for establishing an ontology knowledge base by utilizing multi-modal data mining; and the ontology cognition cultivation module is used for constructing an ontology cognition model and training the ontology cognition model under the guidance of the ontology cognition quantization error and the ontology knowledge base. Compared with the prior art, the method has the advantages that multi-dimensional features of the ontology are represented and perceived, the accuracy of cognitive data is improved, and the overall error is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent agent cognition technology, and in particular to an intelligent agent multi-dimensional ontology incremental cognition system and method. Background Art

[0002] Because of its perception, reasoning and movement capabilities based on embodied interaction, embodied intelligent agents have the potential to move autonomously and stably in open and unstructured environments like humans. Such scenarios full of motion challenges require a higher level of adaptability and intelligence of the intelligent agent's actions. The intelligent agent needs to process the physical interaction between any position of the body and the outside world in real time during autonomous movement to avoid catastrophic motion failures in the operation scenario. At present, in order to effectively represent the topological structure of the intelligent agent to meet the needs of operation and planning, the relationship between each active part is represented by the DH (Denavit-Hartenberg) matrix after being abstracted as virtual rods. By constructing a DH matrix chain (virtual motion rods and joints), the description of the configuration space and motion space of the intelligent agent can be quickly constructed. However, the intelligent agent ontology representation method based on abstract rods is difficult to represent the complex geometric configuration and multi-dimensional physical properties of the intelligent agent. It is still a major challenge to represent and detect the embodied interaction between any position of the body and the external environment and apply it to the motion control of the intelligent agent.

[0003] At present, the main solution to the problem of safe interaction management of the whole body of intelligent agents is to detect interactive collisions based on sensing devices (such as electronic skin, joint torque sensors, motor current feedback). By designing corresponding compliant control methods (impedance control, admittance control), the touch force is controlled within a safe threshold range to avoid catastrophic motion failure. However, this compliant motion control method based on virtual rods lacks consistency with task planning, making it difficult to coordinate the safety of the intelligent agent's actions and the effectiveness of task execution. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an intelligent agent multi-dimensional ontology incremental cognition system and method.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] According to one aspect of the present invention, there is provided an agent multi-dimensional ontology incremental cognition system, characterized in that the system comprises an ontology motion perception module, an ontology cognition evaluation module, a memory pool module, an ontology knowledge mining module and an ontology cognition cultivation module;

[0007] Among them, the ontology motion perception module is used to collect and process the ontology's perception data to obtain multimodal data; the ontology cognition evaluation module is used to use multimodal data to invert the ontology's activities in the digital space, and compare the inversion results with the ontology's cognition results to obtain the ontology's cognition quantization error; the memory pool module is used for data storage; the ontology knowledge mining module is used to use multimodal data to mine the intrinsic relationship between the intelligent agent's perception modality and the ontology's representation dimension, so as to establish an ontology knowledge base as the basis for data extraction and training of the ontology cognition cultivation module; the ontology cognition cultivation module is used to construct an ontology cognition model, and train the ontology cognition model under the guidance of the ontology cognition quantization error and the ontology knowledge base, so as to map the multimodal data to a high-dimensional space to realize the intelligent agent's cognition of the multidimensional characteristics of the ontology.

[0008] As a preferred technical solution, the proprioceptive motion perception module includes a color depth (RGB-Depth Map, RGB-D) visual perception unit, an embodied interaction perception unit and a proprioceptive unit; the RGB-D visual perception unit is used to collect the two-dimensional image and depth information of the intelligent body's proprioceptive motion process observed from its own perspective and transmit it to the visual perception queue; the embodied interaction perception unit is used to collect four modes of temperature, vibration, proximity, and pressure and transmit them to the embodied interaction perception queue; the proprioceptive unit is used to collect the angular displacement, angular velocity and joint torque of the joints during the movement of the intelligent body and transmit them to the proprioceptive queue.

[0009] As a preferred technical solution, the proprioceptive motion perception module also includes a signal unified calibration unit, which is used to adopt a timing consistency calibration algorithm, take the embodied interaction perception signal as the time reference, extract the time samples of corresponding moments from the visual perception queue, the embodied interaction perception queue and the proprioceptive queue respectively, and group the aligned data into action groups to construct visual / tactile-proprioceptive pairs.

[0010] As a preferred technical solution, the ontology cognitive evaluation module includes an ontology action inversion unit, a cognitive bias unit and a data interaction unit; wherein the ontology action inversion unit is used to reproduce the actual limb movements of the intelligent body in the digital space and convert the multimodal information into action-state tensor pairs; the cognitive bias unit uses the inverted actions as a reference for the real actions to calculate the ontology cognitive quantization error; the data interaction unit is used to extract multimodal data and ontology cognitive result data for quantifying the ontology cognitive bias, and to send the ontology cognitive quantization error.

[0011] As a preferred technical solution, the memory pool module includes a short-term memory unit and a long-term memory unit; the short-term memory unit is used to store the multimodal data acquired by the intelligent agent in the current time series, providing a quantitative data source for the multidimensional ontology cognition evaluation module; the long-term memory unit is used to store multimodal data that exceeds the preset tolerance of proprioception deviation recognition, providing training data for the training of the ontology cognition model.

[0012] As a preferred technical solution, the ontology cognition cultivation module includes a forward encoding unit, a reverse encoding unit, a backbone network unit, a forward decoding unit and a reverse decoding unit; wherein, the forward encoding unit includes a visual encoding network built based on a deep residual network (ResNet50) pre-trained network, a proprioceptive encoding network built based on an embedding (Embedding) and multilayer perceptron (Multilayer Perceptron, MLP) network structure, and a tactile encoder network built based on MLP network technology; the reverse encoding unit includes an intelligent body material point set encoding network constructed based on an MLP network; the backbone network includes a multi-layer deep neural network based on a transformer (Transformer) structure, which is used to map multimodal information to a high-dimensional space to implicitly represent the multidimensional motion characteristics of the intelligent body; the forward decoder includes a Transformer and MLP combined network, which is used to output the spatial state of the intelligent body material point set; the reverse decoder includes an MLP network, which is used to output the predicted action of the joint space.

[0013] According to another aspect of the present invention, there is provided an agent multi-dimensional ontology incremental cognitive system, characterized in that a method is applied to the agent multi-dimensional ontology incremental cognitive system as described above, in which a memory pool module is used for data storage, and the method comprises the following steps:

[0014] S1. Using the proprioceptive motion perception module to collect and process the proprioceptive perception data to obtain multimodal data;

[0015] S2. Using the ontology cognition training module, initially training the neural network model based on the multimodal data to establish an ontology cognition model, and using the ontology cognition model to process the multimodal data to obtain the ontology cognition results;

[0016] S3, using the ontology cognition evaluation module, inverting ontology activities in digital space based on multimodal data, and comparing the inversion results with the ontology cognition results to obtain the ontology cognition quantification error;

[0017] S4, using the ontology knowledge mining module to process multimodal data and obtain an ontology knowledge base;

[0018] S5. The ontology cognition quantization error is used to clarify the ontology cognition dimension of the target training, and the knowledge base is used to clarify the perception modality of the target training. Based on the ontology cognition dimension of the target training and the perception modality of the target training, the ontology cognition training module is used to train the ontology cognition model based on multimodal data;

[0019] S6. The trained ontology cognition model processes the multimodal data to output the ontology cognition results, and returns to execute step S4 to realize the multi-dimensional ontology incremental cognition of the intelligent agent.

[0020] As a preferred technical solution, the process of obtaining the ontology cognition quantization error in S3 is: first, the motion state of the intelligent agent in the ontology cognition model is discretized into a finite number of material point sets, and then the visual / tactile-proprioception is extracted from the memory pool module as input to predict the spatial state of these material point sets, so as to reproduce the physical reality in the digital space. Finally, the error of the ontology cognition result is quantified by the mean square error method as the ontology cognition quantization error.

[0021] As a preferred technical solution, S4 uses an ontology knowledge mining module to process multimodal data to obtain a specific process of a knowledge base: first, the perception modality and ontology representation are entityized to obtain entity nodes, and then the multimodal data is differentiated to obtain multimodal data with gradient characteristics, and then the multimodal data with gradient characteristics is fused to the entity nodes to obtain entity data, and based on the Bayesian theorem, the entity data is used to construct an ontology knowledge base.

[0022] As a preferred technical solution, the training process of training the ontology cognition model in S5 adopts Mean Square Error (MSE) as the loss function.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. In the present invention, an incremental ontology cognitive model is established and trained by an intelligent multi-dimensional ontology incremental cognitive system composed of an ontology motion perception module, an ontology cognitive evaluation module, a memory pool module, an ontology knowledge mining module and an ontology cognitive cultivation module. The ontology cognitive model gives the intelligent body the ability to model the multi-dimensional motion characteristics of the whole body. Based on this ability, the intelligent body cognitive model can represent and perceive any position of the body, so that the intelligent body can accurately perceive and represent the multi-dimensional characteristics of the ontology, thereby effectively planning the physical interaction between any position of the body and the outside world.

[0025] 2. In the present invention, the ontology cognitive evaluation module includes an ontology action inversion unit, a cognitive bias unit and a data interaction unit; the intelligent agent inverts the ontology activity in the digital space, and compares the inversion result with the ontology cognitive result to obtain the ontology cognitive quantization error; by observing its own movement in real time and comparing it with the ontology cognitive network result, the ontology cognitive quantization error is obtained, which provides a continuous source of data for the update of the ontology cognitive network and the discovery of new knowledge. By using the ontology cognitive training module to conduct targeted training on data with large cognitive bias, the ontology cognitive network can be continuously improved and enriched during the movement process of the intelligent agent, thereby enhancing its accuracy.

[0026] 3. In the present invention, the intrinsic relationship between the intelligent agent's perception modality and the ontology representation dimension is mined through the ontology knowledge mining module, and the intrinsic relationship between the intelligent agent's perception modality and the ontology representation dimension is mined using multimodal data to establish an ontology knowledge base as the basis for data extraction and training of the ontology cognition cultivation module; so that the ontology cognition network can extract the corresponding perception modality data from the memory pool system in a targeted manner to train the ontology cognition dimension with large error, so that the overall cognitive data error is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a block diagram of the multi-dimensional ontology incremental cognitive system of the present invention;

[0028] Figure 2 The flowchart of the multi-modal proprioceptive motion perception system of the present invention is

[0029] Figure 3 It is a multi-dimensional ontology cognitive evaluation flow chart of the present invention;

[0030] Figure 4 This is a flow chart of the offline and online ontology recognition training of the present invention; DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] Because of its perception, reasoning and movement capabilities based on embodied interaction, embodied intelligent agents have the potential to move autonomously and stably in open and unstructured environments like humans. Such scenarios full of motion challenges require a higher level of adaptability and intelligence of the intelligent agent's actions. The intelligent agent needs to process the physical interaction between any position of the body and the outside world in real time during autonomous movement to avoid catastrophic motion failures in the operation scenario. At present, in order to effectively represent the topological structure of the intelligent agent to meet the needs of operation and planning, the relationship between each active part is represented by the DH matrix after being abstracted as a virtual rod. By constructing a DH matrix chain (virtual motion rods and joints), the description of the configuration space and motion space of the intelligent agent can be quickly constructed. However, the intelligent agent ontology representation method based on abstract rods is difficult to represent the complex geometric configuration and multi-dimensional physical properties of the intelligent agent. It is still a major challenge to represent and detect the embodied interaction between any position of the body and the external environment and apply it to the motion control of the intelligent agent.

[0033] At present, the main solution to the problem of safe interaction management of the whole body of intelligent agents is to detect interactive collisions based on sensing devices (such as electronic skin, joint torque sensors, motor current feedback). By designing corresponding compliant control methods (impedance control, admittance control), the touch force is controlled within a safe threshold range to avoid catastrophic motion failure. However, this compliant motion control method based on virtual rods lacks consistency with task planning, making it difficult to coordinate the safety of the intelligent agent's actions and the effectiveness of task execution.

[0034] In recent years, with the improvement of computing level, machine learning methods, especially deep learning, have outstanding advantages in simulating multi-factor nonlinear strong coupling problems due to their powerful nonlinear fitting ability. Through the high-dimensional space of deep neural networks, the multi-dimensional geometric and physical characteristics of the intelligent body are implicitly represented to directly and digitally represent the intelligent body in a more detailed way, providing the possibility of realizing the whole body motion control and management of the intelligent body. In addition, the intelligent body constantly observes and understands the body movements from its own perspective during the movement, getting rid of the dependence on prior knowledge in the process of motion control, and can adapt to the changes in the body activity diagram in time (parts damage, actuator replacement, etc.). Different from the ontology representation method based on abstract symbols, the intelligent body constantly recognizes the multi-dimensional motion characteristics of the body from the limb activities from its own perspective based on physical reality to establish an intuitive ontology model, which will effectively alleviate the challenges of the whole body interaction planning of the intelligent body.

[0035] Example 1

[0036] In this embodiment, an intelligent multi-dimensional ontology incremental cognitive system is applied. The system structure is as follows: Figure 1As shown, it includes an ontology motion perception module, an ontology cognition evaluation module, a memory pool module, an ontology knowledge mining module and an ontology cognition cultivation module; wherein the ontology motion perception module is used to collect and process the ontology's perception data to obtain multimodal data; the ontology cognition evaluation module is used to use the multimodal data to invert the ontology activity in the digital space, and compare the inversion result with the ontology cognition result to obtain the ontology cognition quantization error; the memory pool module is used for data storage; the ontology knowledge mining module is used to use the multimodal data to mine the intrinsic relationship between the intelligent agent's perception modality and the ontology representation dimension, so as to establish an ontology knowledge base as the basis for data extraction and training of the ontology cognition cultivation module; the ontology cognition cultivation module is used to construct an ontology cognition model, and train the ontology cognition model under the guidance of the ontology cognition quantization error and the ontology knowledge base, so as to map the multimodal data to the high-dimensional space to realize the intelligent agent's cognition of the multidimensional characteristics of the ontology.

[0037] In this embodiment, by establishing a bidirectional ontology cognitive mapping between the joint space and the action space, the intelligent agent is able to predict the form of the entity in the action space based on the joint movements. At the same time, based on the target space state of the entity, the intelligent agent is able to generate joint movements that meet the intentions, and thus can manage embodied interactions at any position of the body.

[0038] Furthermore, by sequentially monitoring and evaluating the deviations between ontology action facts and ontology cognition networks, a steady stream of multimodal ontology perception data will be obtained for sequential increments of ontology cognition.

[0039] At the same time, by mining high-level knowledge of the ontology, the ontology cognitive network will extract data of corresponding modalities for special training, improve training efficiency and reduce engineering workload. The intelligent agent's understanding of its own ontology can become mature and complete in continuous ontology activities.

[0040] In this embodiment, the ontology motion perception module is used to collect and process perception data, and distinguish the intelligent agent from the external environment by identifying the ontology; the ontology cognition evaluation module is used to reproduce the motion of the ontology in the digital space based on the observed multimodal sensor signals, and quantify the error between the ontology cognitive action and the actual action; the memory pool system module is used to collect the robot's motion morphology facts and the ontology motion cognition result data, and provide a data source for the intelligent agent ontology cognition model training and the multi-dimensional ontology cognition evaluation module; the ontology cognition cultivation module is used to implicitly construct a neural network model with a bidirectional mapping from the joint space to the action space, and map the multimodal data to the high-dimensional space to realize the intelligent agent's cognition of the multi-dimensional characteristics of the ontology; the ontology knowledge mining module is used to mine the intrinsic relationship between the intelligent agent's perception modality and the ontology representation dimension, and provide a basis for the ontology cognition cultivation module to extract training data and train the model.

[0041] In this embodiment, the proprioceptive motion perception module collects visual images, proprioception, and tactile data in real time based on the action task and constructs image / tactile-proprioceptive pairs. After the action task is completed, the collected data is packaged and sent to the memory pool module for limb action data storage.

[0042] The multi-dimensional ontology cognition evaluation module extracts multimodal perception data and ontology action cognition result data for quantitative analysis of ontology cognitive bias from the memory pool system, and discretizes the intelligent agent into n The tensor of the material point set, then, quantize the error It is used as data storage basis in the transmission memory pool system.

[0043] In this embodiment, the memory pool system receives the multimodal proprioceptive motion data collected by the multimodal proprioceptive motion perception system and the proprioceptive motion results previewed by the proprioceptive cognition training module, and then stores both into the short-term memory module for use by the multidimensional proprioceptive cognition evaluation module to extract data; at the same time, it receives the quantitative results of the proprioceptive cognition deviation of the multidimensional proprioceptive cognition evaluation module, and when the error Exceeding the tolerance of ontology When the multimodal body action data is transferred from the short-term memory module to the long-term memory module, the memory pool system will delete the data from the short-term memory module.

[0044] In this embodiment, the ontology cognition training module is a bidirectional ontology cognition network from joint space to action space based on the transformer network architecture. The ontology cognition training module receives the action sequence from the planner or the ontology action or action instruction predicted by the target state output. The output result will be transmitted to the memory pool system for short-term memory storage. At the same time, the predicted spatial state is sent to the planner to send the predicted action instruction to the controller of the intelligent body. In the training stage, based on the deviation of ontology cognition in a specific dimension and the modality-dimension mapping map learned by the ontology knowledge mining module 150, matching modality data is extracted from the memory pool system for online training.

[0045] In this embodiment, the ontology knowledge mining module obtains training data from the memory pool system and constructs a modality-relationship-dimension basic knowledge graph unit based on a bottom-up approach; then, the obtained high-level ontology knowledge is transmitted to the ontology cognition cultivation module in a request sending mode for online training.

[0046] In this embodiment, the multimodal proprioceptive motion perception system is as follows: Figure 2 As shown, the specific process is as follows:

[0047] The RGB-D visual perception unit observes the two-dimensional image and depth information of the motion process of the intelligent agent from its own perspective, and the collected signals are sequenced in chronological order and sent to the visual perception queue of the upper computer through the communication link using the right-in and right-out principle;

[0048] The signals collected by the embodied interaction perception unit include four modes: temperature, vibration, proximity, and pressure. The unit is a distributed tactile perception array. The tactile perception unit can transmit signals with other units after encapsulating the data. Multiple units form a tactile perception group. During the movement of the unit robot, the multimodal signals of the intelligent body interacting with the outside world are collected according to the system-specific sampling frequency. Each tactile perception group packages the signals and sends them to the upper computer embodied interaction perception queue using the right-in-right-out principle.

[0049] The proprioceptive unit collects the angular displacement, angular velocity and joint torque of the joints during the movement of the intelligent body to complete the proprioceptive signal observation. The unit is a joint encoder and joint torque sensor distributed in each joint of the intelligent body. After the collected signals are encapsulated into a unified data format, they are transmitted to the proprioceptive queue of the host computer through the communication link using the right-in and right-out principle.

[0050] The signal unified calibration unit adopts a timing consistency calibration algorithm and takes the embodied interaction perception signal as the time reference. It extracts the time samples of the corresponding moments from the visual perception queue, the embodied interaction perception queue and the proprioception queue respectively, and groups the aligned data into action groups to construct visual / tactile-proprioception pairs, forming a multimodal limb action data packet and sending it to the memory pool system.

[0051] In this embodiment, the specific implementation of the multi-dimensional ontology cognitive evaluation module is as follows: Figure 3 As shown, the specific steps are as follows:

[0052] The body motion inversion unit is a deep neural network used for robot body motion recognition. The network discretizes the motion state of the intelligent body into a finite number of material point sets, extracts visual / tactile-proprioception from the memory pool system as input to predict the spatial state of these material point sets, and realizes the reproduction of physical body movements in digital space.

[0053] The cognitive bias assessment unit uses the spatial state of the material point set obtained by the limb movement inversion module as the motion fact, and quantifies the error of the predicted cognitive result of the proprioceptive movement through the mean square error method.

[0054] The data interaction unit opens the communication link between the multi-dimensional proprioceptive cognition evaluation module and the memory pool system based on the communication method sent by the request. When the intelligent agent completes the action task, the data interaction module requests the visual / tactile-proprioceptive pair and proprioceptive cognition result action data from the memory pool system, and sends the received action data to the proprioceptive action inversion module. At the same time, when the cognitive bias module obtains the quantization error, it sends the quantization result to the memory pool system.

[0055] In this embodiment, the specific implementation of the memory pool system is as follows:

[0056] The short-term memory unit stores and forwards the multimodal data generated by performing actions in a short period of time, receives the limb activity data generated by the multimodal proprioceptive motion perception system and the proprioceptive cognition cultivation module, and sends the stored limb activity data according to the request of the multidimensional proprioceptive cognition evaluation module;

[0057] The long-term memory module dynamically stores the limb activity data that exceeds the proprioceptive bias recognition tolerance, providing training data for the proprioceptive cognitive model training; by accepting the proprioceptive cognitive quantization error and limb activity label obtained from the cognitive bias module, the long-term memory module retrieves the limb activity data from the short-term memory module, and when the error exceeds the tolerance interval, the data is stored in this module, otherwise the data is deleted.

[0058] In this embodiment, the specific implementation of the ontology cognition training module is as follows: Figure 4 As shown, the specific process is as follows:

[0059] First, the model is built. The entire network structure consists of five parts: forward encoding, reverse encoding, backbone network, forward decoding, and reverse decoding. Forward encoding includes: a visual encoding module based on the ResNet50 pre-trained network, a proprioceptive encoding module based on the Embedding and MLP network structure, and a tactile encoder module based on the MLP network technology. The reverse encoding module is an encoding module for the set of material points of the intelligent body constructed by the MLP network. The backbone network is a multi-layer deep neural network based on the Transformer structure, which can map multimodal information to a high-dimensional space to implicitly represent the multi-dimensional motion characteristics of the intelligent body. The forward decoder combines the Transformer structure and the MLP structure to output the spatial state of the set of material points of the intelligent body. The reverse decoder is constructed by MLP to output the predicted action of the joint space.

[0060] Then the model offline training process is carried out, and the pre-collected multimodal data set is collected. The model is divided into 80% training set and 20% test set, and training is performed based on the MSE loss function.

[0061] Then deploy the model and migrate the trained model to the intelligent agent computing system.

[0062] Finally, the model enters the online training stage. After the intelligent ontology cognitive model receives the model training task issued by the operator, it retrieves the ontology cognitive quantization error from the memory pool system to clarify the ontology cognitive dimension to be trained. Using the request-response communication mode, the perceptual modality to be trained is obtained from the ontology knowledge mining module, and the training data of the corresponding modality is obtained from the memory pool system for supervised training. The corresponding online training task is completed by evaluating the loss result after model training.

[0063] In this embodiment, the specific implementation of the ontology knowledge mining module is as follows:

[0064] First, the ontology elements are materialized, and the perceptual modality and ontology representation dimensions are materialized into independent entity nodes respectively;

[0065] Then, the structured data is constructed, the time series data is differentiated to obtain multimodal ontological motion data with gradient characteristics, and the data is stored based on entity nodes;

[0066] Finally, ontology knowledge reasoning is performed, and the entity data with gradient features is used to construct a high-level ontology knowledge base of entity-weight-entity based on Bayesian law.

[0067] In summary, this system establishes and trains an incremental ontology cognitive model through an intelligent multi-dimensional ontology incremental cognitive system composed of an ontology motion perception module, an ontology cognitive evaluation module, a memory pool module, an ontology knowledge mining module and an ontology cognitive cultivation module. The ontology cognitive model gives the intelligent agent the ability to model the multi-dimensional motion characteristics of the whole body. Based on this ability, the intelligent agent cognitive model can represent and perceive any position of the body, so that the intelligent agent can accurately perceive and represent the multi-dimensional characteristics of the ontology, thereby effectively planning the physical interaction between any position of the body and the outside world.

[0068] Example 2

[0069] In this embodiment, a multi-dimensional ontology incremental cognition method for an intelligent agent is applied. The method is applied to a system including an ontology motion perception module, an ontology cognition evaluation module, a memory pool module, an ontology knowledge mining module, and an ontology cognition cultivation module. The method uses the memory pool module for data storage. The method includes the following steps:

[0070] S1. Using the proprioceptive motion perception module to collect and process the proprioceptive perception data to obtain multimodal data;

[0071] S2. Using the ontology cognition training module, initially training the neural network model based on the multimodal data to establish an ontology cognition model, and using the ontology cognition model to process the multimodal data to obtain the ontology cognition results;

[0072] S3, using the ontology cognition evaluation module, inverting ontology activities in digital space based on multimodal data, and comparing the inversion results with the ontology cognition results to obtain the ontology cognition quantification error;

[0073] S4, using the ontology knowledge mining module to process multimodal data and obtain an ontology knowledge base;

[0074] S5. The ontology cognition quantization error is used to clarify the ontology cognition dimension of the target training, and the knowledge base is used to clarify the perception modality of the target training. Based on the ontology cognition dimension of the target training and the perception modality of the target training, the ontology cognition training module is used to train the ontology cognition model based on multimodal data;

[0075] S6. The trained ontology cognition model processes the multimodal data to output the ontology cognition results, and returns to execute step S4 to realize the multi-dimensional ontology incremental cognition of the intelligent agent.

[0076] In this method, the process of obtaining the ontology cognition quantization error in S3 is as follows: first, the motion state of the intelligent agent in the ontology cognition model is discretized into a finite number of material point sets, and then the visual / tactile-proprioception is extracted from the memory pool module as input to predict the spatial state of these material point sets, so as to reproduce the physical reality in the digital space. Finally, the error of the ontology cognition result is quantified by the mean square error method as the ontology cognition quantization error.

[0077] In this method, the ontology knowledge mining module is used in S4 to process multimodal data to obtain the specific process of the knowledge base: first, the perception modality and ontology representation are entityized to obtain entity nodes, and then the multimodal data is differentiated to obtain multimodal data with gradient characteristics, and then the multimodal data with gradient characteristics is fused with the entity nodes to obtain entity data, and based on the Bayesian theorem, the entity data is used to construct the ontology knowledge base.

[0078] In this method, the training process of S5 for training the ontology cognition model uses MSE as the loss function

[0079] In this embodiment, the body motion perception module is used to collect and process perception data to distinguish the intelligent body from the external environment; the body motion perception module includes multiple types of perception units and devices, including: RGB-D visual perception unit: used to perform geometric and image perception of the embodied intelligent body to distinguish the body motion from the external environment; embodied interaction perception unit: used to collect object signals at the interaction interface and observe the external physical interaction stimulation received by the intelligent body during movement; proprioception unit: observes the position, speed, and torque signals of the joint space of the intelligent body during movement, and completes the body power unit instruction and state detection; signal unified calibration unit: performs time uniform registration on the collected multi-modal body observation signals to achieve the unification of the sampling frequency and start and end time of the multi-modal perception signals.

[0080] In this embodiment, the ontology cognition evaluation module is used to reproduce the motion of the ontology in the digital space based on the observed multimodal sensor signals to quantify the error between the ontology cognition result and the action fact; the multi-dimensional ontology cognition evaluation module includes multiple submodules, including: an ontology action inversion module: reproducing the limb movements of physical reality in the digital space, and converting the multimodal information into action-state tensor pairs; a cognitive bias slice module: using the inverted action as a reference for the real action, and calculating the cognitive error of the intelligent agent to the ontology cognition result; a data interaction module: extracting the multimodal perception data and ontology cognition result data used for quantifying the ontology cognition bias, and sending the calculation results;

[0081] In this embodiment, the memory pool module is used to store the action form facts of the memory agent, providing a data source for the training of the agent's ontology cognition model and the multi-dimensional ontology cognition evaluation; the memory pool system includes two subsystems, including: a short-term memory module: storing the temporal multimodal data of the robot's past actions in a short period of time, providing a quantitative data source for the multi-dimensional ontology cognition evaluation module; a long-term memory module: storing the limb activity data that exceeds the tolerance of the proprioception deviation recognition, providing training data for the ontology cognition model training;

[0082] In this embodiment, the ontology cognition training module is used to implicitly construct a neural network model for bidirectional mapping from joint space to action space for the intelligent body's physical entity, and the multimodal data is mapped to the high-dimensional space to realize the intelligent body's cognition of the multidimensional characteristics of the ontology; the ontology cognition training module includes two modes: offline and online. The offline mode will target the batch training process of a large amount of perception data, mainly targeting the tasks of the implicit representation training stage of apparent geometry. The online mode targets the multimodal high-dimensional space implicit representation training stage of the physical entity and makes real-time adjustments to the key representation points.

[0083] In this embodiment, the ontology knowledge mining module is used to mine the intrinsic relationship between the intelligent agent perception modality and the ontology representation dimension, which serves as the basis for data extraction and training of the ontology cognition training module.

[0084] In this embodiment, the multimodal proprioceptive motion perception system is as follows: Figure 2 As shown, the specific process is as follows:

[0085] The RGB-D visual perception unit observes the two-dimensional image and depth information of the motion process of the intelligent body from its own perspective, and the collected signals are sequenced in chronological order, and sent to the upper computer visual perception queue through the communication link using the right-in-right-out principle; the embodied interaction perception unit collects signals including four modes: temperature, vibration, proximity, and pressure; this unit is a distributed tactile perception array. The tactile perception unit can transmit signals between other units after encapsulating the data, and multiple units form a tactile perception group; during the movement of the unit robot, the multimodal signals of the intelligent body interacting with the outside world are collected according to the system's specific sampling frequency. Each tactile perception group packages the signals and sends them to the upper computer embodied interaction perception using the right-in-right-out principle. The proprioceptive unit collects the angular displacement, angular velocity and joint torque of the joints during the movement of the intelligent body to complete the proprioceptive signal observation. The unit is a joint encoder and joint torque sensor distributed in each joint of the intelligent body. After the collected signal is encapsulated into a unified data format, it is transmitted to the proprioceptive queue of the host computer through the communication link using the right-in and right-out principle; the signal unified calibration unit adopts a timing consistency calibration algorithm, takes the embodied interaction perception signal as the time reference, and extracts the time samples of the corresponding moments from the visual perception queue, the embodied interaction perception queue and the proprioceptive queue respectively, and the aligned data are grouped into action groups to construct visual / tactile-proprioceptive pairs, forming a multimodal limb action data packet and sending it to the memory pool system.

[0086] In this embodiment, the specific implementation of the multi-dimensional ontology cognitive evaluation module is as follows: Figure 3 As shown, the specific steps are as follows:

[0087] The proprioceptive motion inversion unit is a deep neural network used for robot proprioceptive motion recognition. The network discretizes the motion state of the intelligent body into a finite number of material point sets, extracts visual / tactile-proprioceptive perception from the memory pool system as input to predict the spatial state of these material point sets, and reproduces the physical body movements in the digital space; the cognitive bias assessment unit uses the spatial state of the material point set obtained by the body motion inversion module as the motion fact, and quantifies the error of the predicted proprioceptive motion cognitive result through the mean square error method. The data interaction unit opens the communication link between the multi-dimensional proprioceptive cognition evaluation module and the memory pool system based on the communication method sent by the request. When the intelligent agent completes the action task, the data interaction module requests the visual / tactile-proprioceptive pair and proprioceptive cognition result action data from the memory pool system, and sends the received action data to the proprioceptive action inversion module. At the same time, when the cognitive bias module obtains the quantization error, it sends the quantization result to the memory pool system.

[0088] In this embodiment, the specific implementation of the memory pool system is as follows:

[0089] The short-term memory unit stores and forwards the multimodal data generated by performing actions in a short period of time, receives the limb activity data generated by the multimodal proprioceptive motion perception system and the proprioceptive cognition cultivation module, and sends the stored limb activity data according to the request of the multidimensional proprioceptive cognition evaluation module; the long-term memory module unit dynamically stores the limb activity data that exceeds the proprioceptive deviation recognition tolerance, and provides training data for the proprioceptive cognition model training; by accepting the proprioceptive cognition quantization error and limb activity label obtained from the cognitive deviation module, the long-term memory module retrieves the limb activity data from the short-term memory module, and when the error exceeds the tolerance interval, the data is stored in this module, otherwise the data is deleted.

[0090] In this embodiment, the specific implementation of the ontology cognition training module is as follows: Figure 4 As shown, the specific process is as follows:

[0091] First, the model is built. The entire network structure consists of five parts: forward encoding, reverse encoding, backbone network, forward decoding, and reverse decoding. Forward encoding includes: a visual encoding module built based on the ResNet50 pre-trained network, a proprioceptive encoding module built based on the Embedding and MLP network structure, and a tactile encoder module built based on the MLP network technology. The reverse encoding module is an encoding module for the set of material points of the intelligent body built by the MLP network. The backbone network is a multi-layer deep neural network based on the Transformer structure, which can map multimodal information to a high-dimensional space to implicitly represent the multi-dimensional motion characteristics of the intelligent body. The forward decoder combines the Transformer structure and the MLP structure to output the spatial state of the set of material points of the intelligent body. The reverse decoder is constructed by MLP to output the predicted action of the joint space. Then the model is trained offline to collect the multimodal data set in advance. It is divided into 80% training set and 20% test set, and trained based on the MSE loss function. Then the model is deployed and the trained model is migrated to the intelligent computing system. Finally, the model enters the online training stage. After the intelligent ontology cognitive model receives the model training task issued by the operator, it retrieves the ontology cognitive quantization error from the memory pool system to clarify the ontology cognitive dimension to be trained. Using the request-response communication mode, the perceptual modality to be trained is obtained from the ontology knowledge mining module, and the training data of the corresponding modality is obtained from the memory pool system for supervised training. The corresponding online training task is completed by evaluating the loss result after model training.

[0092] In this embodiment, the specific implementation of the ontology knowledge mining module is as follows:

[0093] First, the ontology elements are materialized, and the perception modality and ontology representation dimension are materialized into independent entity nodes respectively. Then, structured data is constructed, and the time series data is differentiated to obtain multimodal ontology motion data with gradient characteristics, and the data is stored based on entity nodes. Finally, ontology knowledge reasoning is performed, and the entity data with gradient characteristics is used to construct a high-level ontology knowledge base of entity-weight-entity based on Bayesian law.

[0094] In summary, this method enables the intelligent agent to accurately represent and perceive the multi-dimensional characteristics of the ontology. By establishing a two-way ontology cognitive mapping between joint space and action space, the intelligent agent can accurately predict the shape of the ontology in the action space based on the joint action. At the same time, based on the ontology's target space state, the intelligent agent can generate joint actions that meet the intention, and then manage embodied interactions at any position of the body.

[0095] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An intelligent multi-dimensional ontology incremental cognitive system, characterized in that: The system includes an ontology motion perception module, an ontology cognition evaluation module, a memory pool module, an ontology knowledge mining module and an ontology cognition cultivation module; The subject motion perception module is used to collect and process the subject's perception data to obtain multimodal data; The ontology cognition evaluation module is used to invert ontology activities in digital space using multimodal data, and compare the inversion result with the ontology cognition result to obtain the ontology cognition quantization error; The memory pool module is used for data storage; The ontology knowledge mining module is used to use multimodal data to mine the intrinsic relationship between the intelligent agent perception modality and the ontology representation dimension, so as to establish an ontology knowledge base as the basis for data extraction and training of the ontology cognition cultivation module; The ontology cognition training module is used to construct an ontology cognition model and train the ontology cognition model under the guidance of ontology cognition quantization error and ontology knowledge base, so as to map multimodal data to high-dimensional space and realize the intelligent agent's cognition of multidimensional characteristics of ontology.

2. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 1, characterized in that: The proprioceptive motion perception module includes an RGB-D visual perception unit, an embodied interaction perception unit and a proprioceptive perception unit; the RGB-D visual perception unit is used to collect the two-dimensional image and depth information of the proprioceptive motion process observed by the intelligent agent from its own perspective and transmit it to the visual perception queue; the embodied interaction perception unit is used to collect four modes of temperature, vibration, proximity and pressure and transmit them to the embodied interaction perception queue; The proprioceptive unit is used to collect the angular displacement, angular velocity and joint torque of the joints during the movement of the intelligent body and transmit them to the proprioceptive queue.

3. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 2, characterized in that: The proprioceptive motion perception module also includes a signal unified calibration unit, which is used to adopt a timing consistency calibration algorithm, take the embodied interaction perception signal as the time reference, extract the time samples of the corresponding moments from the visual perception queue, the embodied interaction perception queue and the proprioceptive queue respectively, and group the aligned data into action groups to construct visual / tactile-proprioceptive pairs.

4. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 1, characterized in that: The ontology cognitive evaluation module includes an ontology action inversion unit, a cognitive bias unit and a data interaction unit; wherein the ontology action inversion unit is used to reproduce the real body movements of the intelligent body in the digital space and convert the multimodal information into action-state tensor pairs; the cognitive bias unit uses the inverted movements as a reference for the real movements to calculate the ontology cognitive quantization error; the data interaction unit is used to extract multimodal data and ontology cognitive result data for quantifying the ontology cognitive bias, and send the ontology cognitive quantization error.

5. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 1, characterized in that: The memory pool module includes a short-term memory unit and a long-term memory unit; the short-term memory unit is used to store the multimodal data acquired by the intelligent agent in the current time series, providing a quantitative data source for the multidimensional ontology cognition evaluation module; the long-term memory unit is used to store multimodal data that exceeds the preset tolerance of proprioception deviation recognition, providing training data for the training of the ontology cognition model.

6. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 1, characterized in that: The ontology cognition cultivation module includes a forward encoding unit, a reverse encoding unit, a backbone network unit, a forward decoding unit and a reverse decoding unit; wherein, the forward encoding unit includes a visual encoding network built based on a ResNet50 pre-trained network, a proprioceptive encoding network built based on an Embedding and MLP network structure, and a tactile encoder network built based on MLP network technology; the reverse encoding unit includes an intelligent body material point set encoding network built based on an MLP network; the backbone network includes a multi-layer deep neural network based on a Transformer structure, which is used to map multimodal information to a high-dimensional space to implicitly represent the multi-dimensional motion characteristics of the intelligent body; the forward decoder includes a Transformer and MLP combined network, which is used to output the spatial state of the intelligent body material point set; the reverse decoder includes an MLP network, which is used to output the predicted action of the joint space.

7. An intelligent multi-dimensional ontology incremental cognitive system, characterized in that: The method is applied to an intelligent agent multi-dimensional ontology incremental cognitive system as claimed in any one of claims 1 to 6, wherein a memory pool module is used for data storage, and the method comprises the following steps: S1. Using the proprioceptive motion perception module to collect and process the proprioceptive perception data to obtain multimodal data; S2. Using the ontology cognition training module, initially training the neural network model based on the multimodal data to establish an ontology cognition model, and using the ontology cognition model to process the multimodal data to obtain the ontology cognition results; S3, using the ontology cognition evaluation module, inverting ontology activities in digital space based on multimodal data, and comparing the inversion results with the ontology cognition results to obtain the ontology cognition quantification error; S4, using the ontology knowledge mining module to process multimodal data and obtain an ontology knowledge base; S5. The ontology cognition quantization error is used to clarify the ontology cognition dimension of the target training, and the knowledge base is used to clarify the perception modality of the target training. Based on the ontology cognition dimension of the target training and the perception modality of the target training, the ontology cognition training module is used to train the ontology cognition model based on multimodal data; S6. The trained ontology cognition model processes the multimodal data to output the ontology cognition results, and returns to execute step S4 to realize the multi-dimensional ontology incremental cognition of the intelligent agent.

8. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 7, characterized in that: The process of obtaining the ontology cognition quantization error in S3 is as follows: first, the motion state of the intelligent agent in the ontology cognition model is discretized into a finite number of material point sets, and then the visual / tactile-proprioception is extracted from the memory pool module as input to predict the spatial state of these material point sets, so as to reproduce the physical reality in the digital space. Finally, the error of the ontology cognition result is quantified by the mean square error method as the ontology cognition quantization error.

9. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 7, characterized in that: The specific process of using the ontology knowledge mining module to process multimodal data in S4 to obtain the knowledge base is as follows: first, the perception modality and the ontology representation are entityized to obtain entity nodes, and then the multimodal data is differentiated to obtain multimodal data with gradient characteristics, and then the multimodal data with gradient characteristics is fused to the entity nodes to obtain entity data, and based on the Bayesian theorem, the entity data is used to construct an ontology knowledge base.

10. The multi-dimensional ontology incremental cognitive system of an intelligent agent according to claim 7, characterized in that: The training process of training the ontology cognition model in S5 adopts MSE as the loss function.

Images