Robotic control method and system employing mapped visual perception features

Abstract

The application relates to the technical field of robot control, and particularly discloses a robot control method and system adopting a graphed visual perception feature, the method comprising the following steps: collecting images based on an RGB-D camera; performing target identification on the images, positioning a target frame, performing subject identification on the content of the target frame, and determining subject information; constructing a node corresponding to the robot as a reference node, creating a graph containing the reference node according to the subject information, and synchronously calibrating the attributes of each node in the graph; receiving a demand text, identifying the demand text, and generating a control instruction according to the identification result and the graph. The application acquires a visual image, identifies the visual image, positions a subject, synchronously acquires subject information, constructs a graph, takes the graph as information features, generates a control instruction of the robot based on the graph, and in this process, the graph is taken as numerical information and the source of generation of the control instruction, so that the understanding ability of the robot is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a robot control method and system using graphed visual perception features. BACKGROUND

[0002] With the progress of robot technology, it is more and more common to use robots to complete simple and boring tasks. Existing robot control schemes mostly use fixed paths and fixed actions, but in actual scenarios, many tasks are special, which requires robots to have certain "understanding ability". The existing scheme that can provide "understanding ability" is to introduce an AI module on the robot, but this way is too high in cost. How to provide a lower-cost scheme to improve the understanding ability of robots to assist robots in completing simple and boring tasks is a technical problem that the present application technical scheme wants to solve. SUMMARY

[0003] The present application aims to provide a robot control method and system using graphed visual perception features to solve the problems raised in the background art.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0005] A robot control method using graphed visual perception features, the method comprising:

[0006] Upon receiving a graph generation instruction, an image is collected based on an RGB-D camera; the image includes an RGB image and a depth image after time domain matching;

[0007] Target recognition is performed on the image, a target frame is located, subject recognition is performed on the content of the target frame, and subject information is determined; the subject information includes a subject name and a subject position;

[0008] A node corresponding to the robot is constructed as a reference node, a graph containing the reference node is created according to the subject information, and the attributes of each node in the graph are simultaneously calibrated; the graph includes a spatial graph and a semantic graph;

[0009] A demand text is received, the demand text is recognized, and a control instruction is generated according to the recognition result and the graph.

[0010] As a further scheme of the present application, the step of target recognition, target frame positioning, subject recognition on the content of the target frame, and subject information determination comprises:

[0011] An object is framed in the image based on a target detection algorithm to obtain an object frame;

[0012] The object is recognized, and the recognition result is converted into a text as a subject name;

[0013] The depth of each pixel point in the object frame is acquired, and the object position is determined by combining the depth and the image coordinates of the pixel point, as the subject position;

[0014] The subject names and the subject positions in the same image are counted, and the subject information corresponding to the image at the moment is obtained.

[0015] As a further scheme of the present application, the step of creating a graph containing the reference node according to the subject information comprises:

[0016] The robot position is queried, and a node is created according to the robot position, as the reference node;

[0017] A subject node is created according to the subject position, and the subject name is inserted into the subject node as a label;

[0018] The semantic distance between the subject nodes is determined according to the subject names of the subject nodes, edges are generated according to the semantic distance, and a semantic graph is obtained;

[0019] The subject nodes with a semantic distance less than a preset distance threshold are selected, the spatial distance thereof is calculated, the spatial distance between the reference node and each subject node is calculated synchronously, edges are generated according to the spatial distance, and a spatial graph is obtained;

[0020] The subject name is input into a trained attribute analysis model, and the attributes of each node in the image are labeled; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0021] As a further scheme of the present application, the step of creating a graph containing the reference node according to the subject information comprises:

[0022] The spatial graphs of adjacent moments are compared, and the position change vector of each object is determined;

[0023] The position change vector is inserted into the node corresponding to the object as a dynamic attribute.

[0024] As a further scheme of the present application, the step of generating a control instruction according to the recognition result and the graph comprises:

[0025] The demand text is received, the demand text is recognized, and entities and actions are extracted;

[0026] The entities are compared with the subject names of each node in the graph, and a target node is matched;

[0027] Query the attribute of the target node, and determine the execution state of the action according to the attribute; the execution state includes executable and non-executable;

[0028] Select the target node with the execution state as executable, query the subject position corresponding to the target node, and generate a motion instruction;

[0029] When the robot reaches the subject position, an execution instruction is generated based on the action;

[0030] When the execution is completed, a graph update instruction is generated.

[0031] As a further scheme of the application, the method further comprises:

[0032] Read the text recognition result of the demand text, compare the text recognition result, and calculate the text similarity;

[0033] Select the demand text with a text similarity reaching a preset threshold, query the corresponding graph and motion instruction thereof;

[0034] The demand text is taken as a feature, and the motion instruction is taken as a label to train a neural network model; wherein the motion instruction further includes a motion instruction received by the information receiving port;

[0035] The application frequency of the neural network model is adjusted according to the error rate of the neural network model.

[0036] The technical scheme of the application further provides a robot control system using a graphed visual perception feature, the system comprising:

[0037] An image acquisition module is configured to acquire images based on an RGB-D camera when receiving a graph generation instruction; the images include an RGB image and a depth image after time domain matching;

[0038] A subject information determination module is configured to perform target recognition on the images, locate a target frame, perform subject recognition on the content of the target frame, and determine subject information; the subject information includes a subject name and a subject position;

[0039] A graph creation module is configured to construct a node corresponding to the robot as a reference node, create a graph containing the reference node according to the subject information, and synchronize the attributes of each node in the graph; the graph includes a spatial graph and a semantic graph;

[0040] A control instruction generation module is configured to receive a demand text, recognize the demand text, and generate a control instruction according to the recognition result and the graph.

[0041] As a further scheme of the application, the subject information determination module comprises:

[0042] An object frame identification unit is configured to frame an object in an image based on a target detection algorithm to obtain an object frame;

[0043] A subject name generation unit is configured to identify the object and convert the identification result into text as a subject name;

[0044] A subject position acquisition unit is configured to acquire the depth of each pixel point in the object frame, and determine the subject position by combining the depth and the image coordinates of the pixel point as the subject position;

[0045] An information statistics unit is configured to count all subject names and subject positions in the same image to obtain the subject information of the image at the moment.

[0046] As a further scheme of the present application, the atlas creation module comprises:

[0047] A reference node creation unit is configured to query the robot position, and create a node as a reference node according to the robot position;

[0048] A subject node insertion unit is configured to create a subject node according to the subject position, and insert the subject name as a label into the subject node;

[0049] A semantic atlas generation unit is configured to determine the semantic distance between subject nodes according to the subject name of the subject node, generate edges according to the semantic distance, and obtain a semantic atlas;

[0050] A spatial atlas generation unit is configured to select subject nodes with a semantic distance less than a preset distance threshold, calculate the spatial distance thereof, synchronously calculate the spatial distance between the reference node and each subject node, generate edges according to the spatial distance, and obtain a spatial atlas;

[0051] An attribute calibration unit is configured to input the subject name into a trained attribute analysis model to calibrate the attribute of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0052] As a further scheme of the present application, the control instruction generation module comprises:

[0053] A text recognition unit is configured to receive a demand text, recognize the demand text, and extract entities and actions;

[0054] A node matching unit is configured to compare the entities with the subject name of each node in the atlas, and match a target node;

[0055] An execution state determination unit is configured to query the attribute of the target node, and determine the execution state of the action according to the attribute; the execution state comprises executable and non-executable;

[0056] The motion instruction generation unit is configured to select a target node with an executable execution state, query a subject position corresponding to the target node, and generate a motion instruction.

[0057] The execution instruction generation unit is configured to generate an execution instruction based on the action when the robot reaches the subject position.

[0058] The update instruction generation unit is configured to generate a graph update instruction when the execution is completed.

[0059] Compared with the prior art, the present application has the following advantages: the present application acquires a visual image, identifies the visual image, locates a subject, synchronously acquires subject information, constructs a graph, uses the graph as information features, generates a control instruction of a robot based on the graph, and greatly improves the understanding ability of the robot. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0061] Figure 1 The flow block diagram of the robot control method using the graphed visual perception features.

[0062] Figure 2 The first sub-flow block diagram of the robot control method using the graphed visual perception features.

[0063] Figure 3 The second sub-flow block diagram of the robot control method using the graphed visual perception features.

[0064] Figure 4 The third sub-flow block diagram of the robot control method using the graphed visual perception features.

[0065] Figure 5 The composition structure block diagram of the robot control system using the graphed visual perception features. DETAILED DESCRIPTION

[0066] In order to make the technical problems to be solved by the present application, the technical solutions and the beneficial effects more clearly understood, the following will further describe the present application in combination with the drawings and the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0067] Figure 1For the flowchart of the robot control method using the mapping visual perception feature, in the embodiment of the application, a robot control method using a mapping visual perception feature, the method comprises:

[0068] Step S100: when receiving a map generation instruction, acquiring an image based on an RGB-D camera; the image includes an RGB image matched in time domain and a depth image;

[0069] When receiving a map generation instruction, an image is acquired based on an RGB-D camera, the RGB-D camera is installed on a robot or a smart device, and is used to acquire an image stream, wherein an RGB image is a color image, a D image is a depth image, and one RGB image and one D image are a set of inputs; it is worth mentioning that the acquired image is generally an image sequence, rather than a single image, because the volatility of a single image is too large, and the analysis process is not universal, but in order to simplify the description process, the processing process of only one image is described in the technical solution of the application, in actual application, the processing process of each image needs to be applied multiple times, all images are processed, and finally the processing results are comprehensively processed to determine a final processing result, that is, a control instruction.

[0070] Step S200: target recognition is performed on the image, a target frame is located, subject recognition is performed on the content of the target frame, and subject information is determined; the subject information includes a subject name and a subject position;

[0071] The perception objects in the acquired image are normalized to remove redundant information and integrate similar targets, and a compact "object representation set" is generated, which provides clear node definition for subsequent map modeling, and the specific operation process includes:

[0072] 1. A depth detection model (such as YOLOv5, YOLOv7, YOLO-NAS, DETR or FasterR-CNN) is used to detect all targets on each frame of image.

[0073] 2. Non-maximum suppression (NMS) is applied to remove redundant detection frames that are too overlapped in the detection process.

[0074] 3. According to the position and semantic category of the target, DBSCAN (density-based clustering algorithm) is used to cluster the target, and repeated expression is eliminated to make the data more compact.

[0075] 4. Depth semantic features are extracted for each target: the image area is cropped using the target frame, and the image area is encoded into a feature vector through a feature encoding network (such as ResNet18+Transformer encoder), which is convenient for subsequent processing.

[0076] In a word, the step S200 is used for locating the object in the image and identifying the object to obtain the subject name and the subject position, wherein the subject name is a word vector, and the subject position is determined by the pixel position and the depth of the image.

[0077] Step S300: constructing a node corresponding to the robot as a reference node, creating a graph containing the reference node according to the subject information, and synchronously calibrating the attributes of each node in the graph; the graph includes a spatial graph and a semantic graph;

[0078] After obtaining the subject information, the graph is created in combination with the information of the robot itself, the graph is a graph structure including nodes and edges, each node corresponds to a subject, and the node corresponding to the robot is called a reference node; at the same time, the attributes of each node in the graph also need to be calibrated, in fact, the subject name is also one of the attributes, but it is more important, so it is listed as a label.

[0079] It should be noted that the graph constructed in the technical scheme of the application includes two kinds, one is a spatial graph, and the other is a semantic graph, the spatial graph generates edges based on spatial distance, and the semantic graph generates edges based on semantic distance, the spatial graph reflects the distance of each subject in space, and the semantic graph reflects the distance of each subject in semantic space, the closer, the more likely the two subjects are to have a relationship, this process is the core of the technical scheme of the application, the object set after semantic compression is structured into a graph structure, so that the system has the modeling ability of "space + semantics + context", and provides structural cognitive ability for subsequent behavior prediction and task planning.

[0080] Step S400: receiving a demand text, identifying the demand text, and generating a control instruction according to the identification result and the graph;

[0081] In the robot control stage, the demand text such as "grabbing the water cup on the table" is received, the demand text is identified to identify the key words and actions, and then the control instruction is generated in combination with the graph, the control instruction is used to control the robot to complete the demand text, including reaching the specified position and completing the corresponding action, therefore, the control instruction actually includes a motion instruction and an action execution instruction.

[0082] As a preferred embodiment of the technical scheme of the application, a feasible data structure and its application mode are provided to complete the above steps, which are as follows:

[0083] I. Input image:

[0084] The system obtains an environment image stream through an RGB-D camera (such as RealSense D435) installed on a robot or intelligent device. Each frame of image contains an RGB image (color) and a D image (depth), forming a set of perception inputs.

[0085] II. Semantic Compression (Target Detection + Semantic Clustering):

[0086] 1) Purpose of Execution:

[0087] The semantic-level normalized expression of the perception objects in the original image is performed, redundant detection boxes are removed, and semantically similar targets are merged, outputting a compact "object representation set" that provides clear node definitions for graph modeling.

[0088] 2) Input Description: Input image frame: I ∈ R^{H×W×3} (RGB image); Pre-processed image: can be scaled and normalized in size; Depth map (optional): D ∈ R^{H×W} (from RGB-D camera).

[0089] 3) Core Processing Steps:

[0090] Target detection, using a lightweight or medium-sized deep detection model (one of the options):

[0091] YOLOv5 / YOLOv7 / YOLO-NAS (suitable for embedded), DETR (based on Transformer, supports semantic embedding), and Faster R-CNN (high precision, slow speed).

[0092] Redundant target filtering, applying non-maximum suppression (NMS) to remove overlapping boxes:

[0093] Spatial semantic clustering, clustering the remaining targets according to spatial location and semantic category to eliminate duplicate expressions:

[0094] Using DBSCAN (Density-Based Spatial Clustering).

[0095] Feature extraction, extracting a depth semantic feature vector for each target o_i (for subsequent graph construction):

[0096] Using the image region box_i for cropping, inputting a feature encoding network (such as ResNet18 + Transformer encoder), and outputting a feature vector.

[0097] III. Graph Construction (GNN Modeling):

[0098] 1) Execution purpose: structure the semantic compressed object set into a graph structure, so that the system has the modeling ability of "space + semantics + context", and provides structural cognitive ability for subsequent behavior prediction and task planning.

[0099] 2) Graph structure definition: construct graph G=(V, E, A): V: node set (each object is a node); E: edge set (relationship between nodes); A: attribute set (additional information of nodes and edges).

[0100] 3) Node construction method: each node v_i corresponds to semantic object o_i', containing the following attributes:

[0101]

[0102] Attribute table

[0103] 4) Edge construction method (E): edges represent semantic or spatial relationships between two nodes, and the construction rules are as follows:

[0104] Edge establishment condition: recommended parameters: T_dist=1m (actual space), T_sim=0.8 and T_depth=0.5m.

[0105] 5) Graph neural network modeling (graph feature fusion), using GAT (Graph Attention Network) structure to aggregate node features:

[0106] W: learnable linear transformation matrix;

[0107] a^T: attention score parameter;

[0108] h_i: input node feature, h_i': output fusion feature.

[0109] 6) Graph output structure, the final output structure is scene graph:

[0110] G=(V, E, A);

[0111] Each node v_i: entity object with semantic context within the graph;

[0112] Each edge e_ij: spatial / semantic relationship between entities;

[0113] Can be used in subsequent modules: behavior prediction, path planning, semantic question answering, etc.

[0114] Four, cognitive decision (task recognition + path planning):

[0115] Execution method: input the graph G into the decision engine, identify the task intent and plan the action path or execute the logic.

[0116] Task recognition: use subgraph matching algorithm (such as VF2 or Graph Matching Network) to identify target scene structure; task matching such as: "object A belongs to region B", "person close to table C", etc.

[0117] Behavior planning algorithm: path type task: use A* or Dijkstra shortest path algorithm;

[0118] Multi-step task sequence: use reinforcement learning (DQN, DDPG, etc.).

[0119] Five, control output (action execution):

[0120] Execution mode:

[0121] Encode the above decision results into control instructions and send them to the robot execution unit (such as mobile chassis, robot arm controller).

[0122] Communication method: ROS Topic or Service, serial control or WebSocket / API interface.

[0123] Figure 2 The first sub-flow block diagram of the robot control method using the graph-based visual perception feature, the steps of the target recognition, positioning the target frame, and subject recognition of the content of the target frame to determine the subject information include:

[0124] Step S201: frame the object in the image based on the target detection algorithm, and obtain the object frame;

[0125] Step S202: identify the object and convert the identification result into text as the subject name;

[0126] Step S203: obtain the depth of each pixel point in the object frame, and determine the object position by combining the depth and the image coordinates of the pixel point, as the subject position;

[0127] Step S204: count all subject names and subject positions in the same image to obtain the subject information corresponding to the image at the moment.

[0128] The above content limits the identification process of the image, and the object in the image is framed based on a target detection algorithm to obtain an object frame. There are many target detection algorithms, such as a deep detection model, including YOLOv5, YOLOv7, YOLO-NAS, DETR or Faster R-CNN. After locating the target, the target is framed to obtain the object frame. Then, the object is identified, and the existing object identification algorithm is applied. The object identification result is converted into text as the subject name. Then, the depth of each pixel point in the object frame is obtained, and the depth and the image coordinates of the pixel point are combined to determine the object position as the subject position. Finally, all subject names and their subject positions in the same image are counted to obtain the subject information of the image at the moment.

[0129] Figure 3 The second sub-flow chart of the robot control method using the graph-based visual perception feature is provided, the node corresponding to the robot is constructed as a reference node, and the graph containing the reference node is created according to the subject information. The steps of synchronously calibrating the attributes of each node in the graph include:

[0130] Step S301: querying the robot position, creating a node according to the robot position as a reference node;

[0131] Step S302: creating a subject node according to the subject position, and inserting the subject name as a label into the subject node;

[0132] Step S303: determining the semantic distance between the subject nodes according to the subject name of the subject node, generating edges according to the semantic distance, and obtaining a semantic graph;

[0133] Step S304: selecting the subject nodes with a semantic distance less than a preset distance threshold, calculating the spatial distance thereof, synchronously calculating the spatial distance between the reference node and each subject node, generating edges according to the spatial distance, and obtaining a spatial graph;

[0134] Step S305: inputting the subject name into a trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0135] In one example of the technical scheme of the present application, the robot position is queried, a node is created according to the robot position as a reference node, and a subject node is created according to the subject position. The subject name is inserted into the subject node as a label. On this basis, the semantic distance between the subject nodes is determined according to the subject name of the subject node, edges are generated according to the semantic distance, and a semantic graph is obtained. The semantic distance can convert the subject name into a word vector, and then the distance between the word vectors is taken as the semantic distance. Edges are generated according to the semantic distance, and a semantic graph is obtained.

[0136] After the semantic graph is generated, main body nodes with a semantic distance less than a preset distance threshold are selected, and the smaller the semantic distance, the more likely the two main body nodes appear in the same demand text, at this time, the spatial distance between them is obtained, which also means that not all main body nodes need to obtain the spatial distance; in addition, for the robot, it needs to calculate the spatial distance between each main body node, and thus, an edge is generated according to the calculated spatial distance to obtain a spatial graph.

[0137] Finally, the main body name is input into the trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value, in simple terms, the attribute analysis model is a database with data retrieval function.

[0138] As a preferred embodiment of the technical scheme of the application, the step of constructing a node corresponding to the robot as a reference node, creating a graph containing the reference node according to the main body information, and synchronously calibrating the attributes of each node in the graph further comprises:

[0139] Comparing the spatial graphs of adjacent time points to determine the position change vector of each object;

[0140] Inserting the position change vector as a dynamic attribute into the node corresponding to the object.

[0141] In an example of the technical scheme of the application, the content of the attribute is expanded, the spatial graphs of adjacent time points are compared to determine the position change vector of each object, the position change vector represents the speed in this period of time, and the position change vector is inserted as a dynamic attribute into the node corresponding to the object, at this time, the attribute contains a parameter reflecting the motion state of the object.

[0142] Figure 4 For the third sub-flow block diagram of the robot control method using the graph-based visual perception feature, the step of receiving a demand text, identifying the demand text, and generating a control instruction according to the identification result and the graph comprises:

[0143] Step S401: receiving a demand text, identifying the demand text, and extracting entities and actions;

[0144] Step S402: comparing the entities with the main body names of each node in the graph to match target nodes;

[0145] Step S403: querying the attributes of the target nodes and determining the execution state of the actions according to the attributes; the execution state includes executable and non-executable;

[0146] Step S404: selecting target nodes with an executable execution state, querying the main body positions corresponding to the target nodes, and generating motion instructions;

[0147] Step S405: when the robot reaches the subject position, generating an execution instruction based on the action;

[0148] Step S406: when the execution is completed, generating a graph update instruction.

[0149] In an example of the technical scheme of the present application, the generation process of the control instruction is described, the demand text is received, the entity and the action are extracted from the demand text, the entity is also a text, and the value space of the entity is the same as the value space of the subject name, the entity is compared with the subject name of each node in the graph, the target node is matched, the attribute of the target node is queried, the execution state of the action is determined according to the attribute, for example, the attribute of the object is a fixed object, when the execution state is grabbing, the execution state is not executable, the target node with the executable execution state is selected, the subject position corresponding to the target node is queried, the motion instruction along the path is generated, this process can be assisted by A* or Dijkstra shortest path algorithm, the focus is to generate the motion instruction along the path, when the robot reaches the subject position, the execution instruction is generated based on the action, when the execution is completed, the graph update instruction is generated.

[0150] For the graph update instruction, it is the same as the received graph generation instruction in step S100, thus, the technical scheme of the present application is in a cyclic execution state, and the control process of the robot itself is a cyclic state.

[0151] As a preferred embodiment of the technical scheme of the present application, the method further comprises:

[0152] reading the text recognition result of the demand text, comparing the text recognition result, and calculating the text similarity;

[0153] selecting the demand text with the text similarity reaching a preset threshold, querying the corresponding graph and motion instruction thereof;

[0154] taking the demand text as a feature and taking the motion instruction as a label to train a neural network model; wherein the motion instruction further comprises a motion instruction received by the information receiving port;

[0155] adjusting the application frequency of the neural network model according to the error rate of the neural network model.

[0156] In an example of the technical solution of the present application, a more simplified control instruction generation process is provided, in actual application, when the robot is used to work, the work content is almost repetitive, such as material grabbing and distribution in a workshop, which needs to be repeated, and the robot only performs such work for a long time, in addition, the number of work types is also small, which means that the work content of the robot is limited to a small range, on this basis, the text recognition result of the demand text is read, the text recognition result is compared, the text similarity is calculated, the demand text whose text similarity reaches the preset threshold is selected, the corresponding graph and motion instruction of the demand text is queried, thereby, the graph and motion instruction of the same demand (or very similar demand) are classified into a category, the graph and motion instruction of these demands are counted, the demand text is taken as a feature, and the motion instruction is taken as a label, and a neural network model is trained, at this time, each demand can obtain a neural network model, in actual application, the neural network model is queried according to the demand text, and the control instruction can be directly generated according to the actually constructed graph, because the sample value of the graph-control instruction is limited.

[0157] In addition, in the application process, the error rate of the neural network model also needs to be calculated in real time, and the application frequency of the neural network model is adjusted according to the error rate, the application frequency is how many times in a time period, and the application frequency is inversely proportional to the error rate.

[0158] Figure 5 For the composition structure block diagram of the robot control system adopting the graph-based visual perception feature, in the embodiment of the present application, a robot control system adopting a graph-based visual perception feature, the system 10 comprises:

[0159] An image acquisition module 11 is configured to acquire images based on an RGB-D camera when receiving a graph generation instruction; the images comprise an RGB image and a depth image after time domain matching;

[0160] A main body information determination module 12 is configured to perform target recognition on the images, locate a target frame, perform main body recognition on the content of the target frame, and determine main body information; the main body information comprises a main body name and a main body position;

[0161] A graph creation module 13 is configured to construct a node corresponding to the robot as a reference node, create a graph containing the reference node according to the main body information, and synchronize the attributes of each node in the graph; the graph comprises a spatial graph and a semantic graph;

[0162] A control instruction generation module 14 is configured to receive a demand text, recognize the demand text, and generate a control instruction according to the recognition result and the graph.

[0163] Further, the subject information determination module 12 comprises:

[0164] An object frame identification unit is configured to frame an object in an image based on a target detection algorithm to obtain an object frame.

[0165] A subject name generation unit is configured to identify the object and convert the identification result into text as a subject name.

[0166] A subject position acquisition unit is configured to acquire the depth of each pixel point in the object frame and determine the object position as a subject position by combining the depth and the image coordinates of the pixel point.

[0167] An information statistics unit is configured to count all subject names and their subject positions in the same image to obtain the subject information of the image at the moment.

[0168] Specifically, the graph creation module 13 comprises:

[0169] A reference node creation unit is configured to query the robot position and create a node as a reference node according to the robot position.

[0170] A subject node insertion unit is configured to create a subject node according to the subject position and insert the subject name as a label into the subject node.

[0171] A semantic graph generation unit is configured to determine the semantic distance between subject nodes according to the subject names of the subject nodes, generate edges according to the semantic distance, and obtain a semantic graph.

[0172] A spatial graph generation unit is configured to select subject nodes with a semantic distance less than a preset distance threshold, calculate the spatial distance thereof, synchronously calculate the spatial distance between the reference node and each subject node, generate edges according to the spatial distance, and obtain a spatial graph.

[0173] An attribute calibration unit is configured to input the subject name into a trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0174] Further, the control instruction generation module 14 comprises:

[0175] A text recognition unit is configured to receive a demand text, recognize the demand text, and extract entities and actions.

[0176] A node matching unit is configured to compare the entities with the subject names of each node in the graph to match a target node.

[0177] An execution state determination unit is configured to query the attributes of the target node and determine the execution state of the action according to the attributes; the execution state comprises executable and non-executable.

[0178] a motion instruction generation unit configured to select a target node with an executable execution state, query a subject position corresponding to the target node, and generate a motion instruction;

[0179] an execution instruction generation unit configured to generate an execution instruction based on the action when the robot reaches the subject position;

[0180] an update instruction generation unit configured to generate a graph update instruction when the execution is completed.

[0181] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot control method employing a mapping visual perception feature, characterized by, The method comprises: Upon receiving a graph generation instruction, acquiring an image based on an RGB-D camera; the image comprises an RGB image and a depth image after time domain matching; Performing target recognition on the image, locating a target frame, performing subject recognition on the content of the target frame, and determining subject information; the subject information comprises a subject name and a subject position; Constructing a node corresponding to the robot as a reference node, creating a graph comprising the reference node based on the subject information, and synchronously calibrating the attributes of each node in the graph; the graph comprises a spatial graph and a semantic graph; Receiving a demand text, performing recognition on the demand text, and generating a control instruction based on the recognition result and the graph; The step of constructing a node corresponding to the robot as a reference node, creating a graph comprising the reference node based on the subject information, and synchronously calibrating the attributes of each node in the graph comprises: Querying the position of the robot, creating a node based on the position of the robot as a reference node; Creating a subject node based on the subject position and inserting the subject name as a label into the subject node; Determining the semantic distance between the subject nodes based on the subject name of the subject node, generating edges based on the semantic distance, and obtaining a semantic graph; Selecting subject nodes with a semantic distance less than a preset distance threshold, calculating the spatial distance thereof, synchronously calculating the spatial distance between the reference node and each subject node, generating edges based on the spatial distance, and obtaining a spatial graph; Inputting the subject name into a trained attribute analysis model to calibrate the attributes of each node in the image; the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value; The step of constructing a node corresponding to the robot as a reference node, creating a graph comprising the reference node based on the subject information, and synchronously calibrating the attributes of each node in the graph further comprises: Comparing the spatial graphs of adjacent time points to determine the position change vector of each object; Inserting the position change vector as a dynamic attribute into the node corresponding to the object.

2. The robot control method employing mapped visual perception features according to claim 1, wherein, The step of performing target recognition on the image, locating a target frame, performing subject recognition on the content of the target frame, and determining subject information comprises: Based on a target detection algorithm, frame the object in the image to obtain an object frame; Identify the object and convert the recognition result into a text as a subject name; Obtain the depth of each pixel point in the object frame, and determine the object position based on the depth and the image coordinates of the pixel point as a subject position; Statistically obtain all subject names and subject positions in the same image to obtain the subject information of the time point corresponding to the image.

3. The method of robot control with mapped visual perception features of claim 1, wherein, The step of receiving a demand text, performing recognition on the demand text, and generating a control instruction based on the recognition result and the graph comprises: Receiving a demand text, performing recognition on the demand text, extracting entities and actions; Comparing the entities with the subject names of each node in the graph to match target nodes; Querying the attributes of the target nodes to determine the execution state of the actions; the execution state comprises executable and non-executable; Selecting target nodes with an executable execution state, querying the subject position corresponding to the target nodes, and generating a motion instruction; When the robot reaches the subject position, generating an execution instruction based on the action; When the execution is completed, generating a graph update instruction.

4. The method of claim 1, wherein the method further comprises: The method further comprises: reading a text recognition result of the demand text, comparing the text recognition result, and calculating a text similarity; selecting the demand text with a text similarity reaching a preset threshold, querying a corresponding graph and motion instruction thereof; training a neural network model by taking the demand text as a feature and the motion instruction as a label; wherein the motion instruction further comprises a motion instruction received by an information receiving port; adjusting an application frequency of the neural network model according to an error rate of the neural network model.

5. A robot control system employing mapped visual perception features, characterized by, The system comprises: an image acquisition module configured to acquire an image based on an RGB-D camera when a graph generation instruction is received; the image comprises an RGB image and a depth image after time domain matching; a main body information determination module configured to perform target recognition on the image, locate a target frame, perform main body recognition on a content of the target frame, and determine main body information; the main body information comprises a main body name and a main body position; a graph creation module configured to construct a node corresponding to the robot as a reference node, create a graph comprising the reference node according to the main body information, and simultaneously calibrate attributes of each node in the graph; the graph comprises a spatial graph and a semantic graph; a control instruction generation module configured to receive a demand text, recognize the demand text, and generate a control instruction according to a recognition result and the graph; the graph creation module comprises: a reference node creation unit configured to query a robot position, create a node according to the robot position, and take the node as a reference node; a main body node insertion unit configured to create a main body node according to a main body position, and insert a main body name as a label into the main body node; a semantic graph generation unit configured to determine a semantic distance between main body nodes according to the main body names of the main body nodes, generate an edge according to the semantic distance, and obtain a semantic graph; a spatial graph generation unit configured to select main body nodes with a semantic distance less than a preset distance threshold, calculate a spatial distance thereof, simultaneously calculate spatial distances between the reference node and each main body node, generate an edge according to the spatial distances, and obtain a spatial graph; an attribute calibration unit configured to input the main body name into a trained attribute analysis model, and calibrate attributes of each node in the image; wherein the attribute analysis model is a data query model, and an attribute of each text is a pre-stored value; the content of constructing the node corresponding to the robot as the reference node, creating the graph comprising the reference node according to the main body information, and simultaneously calibrating the attributes of each node in the graph further comprises: comparing spatial graphs of adjacent time points to determine a position change vector of each object; inserting the position change vector as a dynamic attribute into a node corresponding to the object.

6. The robotic control system employing mapped visual perception features of claim 5, wherein, the main body information determination module comprises: an object frame recognition unit configured to frame an object in the image based on a target detection algorithm to obtain an object frame; a main body name generation unit configured to recognize the object, and convert a recognition result into a text as a main body name; a main body position acquisition unit configured to acquire depths of each pixel point in the object frame, and determine an object position as a main body position by combining the depths and image coordinates of the pixel points; an information statistics unit configured to count all main body names and main body positions in a same image to obtain main body information of a time point corresponding to the image.

7. The robotic control system employing mapped visual perception features of claim 5, wherein, The control instruction generation module comprises: A text recognition unit configured to receive a demand text, recognize the demand text, and extract entities and actions; A node matching unit configured to compare the entities with subject names of each node in a graph, and match target nodes; An execution state determination unit configured to query attributes of the target nodes, and determine execution states of the actions according to the attributes; the execution states include executable and unexecutable; A motion instruction generation unit configured to select the target nodes with the execution states being executable, query subject positions corresponding to the target nodes, and generate motion instructions; An execution instruction generation unit configured to generate execution instructions based on the actions when the robot reaches the subject positions; An update instruction generation unit configured to generate graph update instructions when the execution is completed.

Images