Robot control method and system adopting mapping visual perception characteristics

Abstract

The invention relates to the technical field of robot control, and particularly discloses a robot control method and system adopting mapping visual perception characteristics, and the method comprises the steps: collecting an image based on an RGB-D camera; performing target identification on the image, positioning a target frame, performing subject identification on the content of the target frame, and determining subject information; nodes corresponding to the robot are constructed as reference nodes, a map containing the reference nodes is created according to the subject information, and attributes of all the nodes in the map are synchronously calibrated; and receiving a demand text, identifying the demand text, and generating a control instruction according to an identification result and the map. According to the method, the visual image is acquired, the visual image is identified, the main body is positioned, the main body information is synchronously acquired, the atlas is constructed, the atlas is used as information characteristics, the control instruction of the robot is generated based on the atlas, in the process, the atlas is used as numerical information and a generation source of the control instruction, and the understanding ability of the robot is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a robot control method and system using graphed visual perception features. BACKGROUND

[0002] With the progress of robot technology, it is more and more common to use robots to complete simple and boring tasks. Existing robot control schemes mostly use fixed paths and fixed actions, but in actual scenarios, many tasks are special, which requires robots to have certain "understanding ability". The existing scheme that can provide "understanding ability" is to introduce an AI module on the robot, but this way is too high in cost. How to provide a lower-cost scheme to improve the understanding ability of robots to assist robots in completing simple and boring tasks is a technical problem that the present application technical scheme wants to solve. SUMMARY

[0003] The present application aims to provide a robot control method and system using graphed visual perception features to solve the problems raised in the background art.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0005] A robot control method using graphed visual perception features, the method comprising:

[0006] Upon receiving a graph generation instruction, an image is collected based on an RGB-D camera; the image includes an RGB image and a depth image after time domain matching;

[0007] Target recognition is performed on the image, a target frame is located, subject recognition is performed on the content of the target frame, and subject information is determined; the subject information includes a subject name and a subject position;

[0008] A node corresponding to the robot is constructed as a reference node, a graph containing the reference node is created according to the subject information, and the attributes of each node in the graph are simultaneously calibrated; the graph includes a spatial graph and a semantic graph;

[0009] A demand text is received, the demand text is recognized, and a control instruction is generated according to the recognition result and the graph.

[0010] As a further scheme of the present application, the step of target recognition, target frame positioning, subject recognition on the content of the target frame, and subject information determination comprises:

[0011] An object is framed in the image based on a target detection algorithm to obtain an object frame;

[0012] The object is recognized, and the recognition result is converted into a text as a subject name;

[0013] The depth of each pixel point in the object frame is acquired, and the object position is determined by combining the depth and the image coordinates of the pixel point, as the subject position;

[0014] The subject names and the subject positions in the same image are counted, and the subject information corresponding to the image at the moment is obtained.

[0015] As a further scheme of the present application, the step of creating a graph containing the reference node according to the subject information comprises:

[0016] The robot position is queried, and a node is created according to the robot position, as the reference node;

[0017] A subject node is created according to the subject position, and the subject name is inserted into the subject node as a label;

[0018] The semantic distance between the subject nodes is determined according to the subject names of the subject nodes, edges are generated according to the semantic distance, and a semantic graph is obtained;

[0019] The subject nodes with a semantic distance less than a preset distance threshold are selected, the spatial distance thereof is calculated, the spatial distance between the reference node and each subject node is calculated synchronously, edges are generated according to the spatial distance, and a spatial graph is obtained;

[0020] The subject name is input into a trained attribute analysis model, and the attributes of each node in the image are labeled; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0021] As a further scheme of the present application, the step of creating a graph containing the reference node according to the subject information comprises:

[0022] The spatial graphs of adjacent moments are compared, and the position change vector of each object is determined;

[0023] The position change vector is inserted into the node corresponding to the object as a dynamic attribute.

[0024] As a further scheme of the present application, the step of generating a control instruction according to the recognition result and the graph comprises:

[0025] The demand text is received, the demand text is recognized, and entities and actions are extracted;

[0026] The entities are compared with the subject names of each node in the graph, and a target node is matched;

[0027] Query the attribute of the target node, and determine the execution state of the action according to the attribute; the execution state includes executable and non-executable;

[0028] Select the target node with the execution state as executable, query the subject position corresponding to the target node, and generate a motion instruction;

[0029] When the robot reaches the subject position, an execution instruction is generated based on the action;

[0030] When the execution is completed, a graph update instruction is generated.

[0031] As a further scheme of the application, the method further comprises:

[0032] Read the text recognition result of the demand text, compare the text recognition result, and calculate the text similarity;

[0033] Select the demand text with a text similarity reaching a preset threshold, query the corresponding graph and motion instruction thereof;

[0034] The demand text is taken as a feature, and the motion instruction is taken as a label to train a neural network model; wherein the motion instruction further includes a motion instruction received by the information receiving port;

[0035] The application frequency of the neural network model is adjusted according to the error rate of the neural network model.

[0036] The technical scheme of the application further provides a robot control system using a graphed visual perception feature, the system comprising:

[0037] An image acquisition module is configured to acquire images based on an RGB-D camera when receiving a graph generation instruction; the images include an RGB image and a depth image after time domain matching;

[0038] A subject information determination module is configured to perform target recognition on the images, locate a target frame, perform subject recognition on the content of the target frame, and determine subject information; the subject information includes a subject name and a subject position;

[0039] A graph creation module is configured to construct a node corresponding to the robot as a reference node, create a graph containing the reference node according to the subject information, and synchronize the attributes of each node in the graph; the graph includes a spatial graph and a semantic graph;

[0040] A control instruction generation module is configured to receive a demand text, recognize the demand text, and generate a control instruction according to the recognition result and the graph.

[0041] As a further scheme of the application, the subject information determination module comprises:

[0042] An object frame identification unit is configured to frame an object in an image based on a target detection algorithm to obtain an object frame;

[0043] A subject name generation unit is configured to identify the object and convert the identification result into text as a subject name;

[0044] A subject position acquisition unit is configured to acquire the depth of each pixel point in the object frame, and determine the subject position by combining the depth and the image coordinates of the pixel point as the subject position;

[0045] An information statistics unit is configured to count all subject names and subject positions in the same image to obtain the subject information of the image at the moment.

[0046] As a further scheme of the present application, the atlas creation module comprises:

[0047] A reference node creation unit is configured to query the robot position, and create a node as a reference node according to the robot position;

[0048] A subject node insertion unit is configured to create a subject node according to the subject position, and insert the subject name as a label into the subject node;

[0049] A semantic atlas generation unit is configured to determine the semantic distance between subject nodes according to the subject name of the subject node, generate edges according to the semantic distance, and obtain a semantic atlas;

[0050] A spatial atlas generation unit is configured to select subject nodes with a semantic distance less than a preset distance threshold, calculate the spatial distance thereof, synchronously calculate the spatial distance between the reference node and each subject node, generate edges according to the spatial distance, and obtain a spatial atlas;

[0051] An attribute calibration unit is configured to input the subject name into a trained attribute analysis model to calibrate the attribute of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0052] As a further scheme of the present application, the control instruction generation module comprises:

[0053] A text recognition unit is configured to receive a demand text, recognize the demand text, and extract entities and actions;

[0054] A node matching unit is configured to compare the entities with the subject name of each node in the atlas, and match a target node;

[0055] An execution state determination unit is configured to query the attribute of the target node, and determine the execution state of the action according to the attribute; the execution state comprises executable and non-executable;

[0056] The motion instruction generation unit is configured to select a target node with an executable execution state, query a subject position corresponding to the target node, and generate a motion instruction.

[0057] The execution instruction generation unit is configured to generate an execution instruction based on the action when the robot reaches the subject position.

[0058] The update instruction generation unit is configured to generate a graph update instruction when the execution is completed.

[0059] Compared with the prior art, the present application has the following advantages: the present application acquires a visual image, identifies the visual image, locates a subject, synchronously acquires subject information, constructs a graph, uses the graph as information features, generates a control instruction of a robot based on the graph, and greatly improves the understanding ability of the robot. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0061] Figure 1 The flow block diagram of the robot control method using the graphed visual perception features.

[0062] Figure 2 The first sub-flow block diagram of the robot control method using the graphed visual perception features.

[0063] Figure 3 The second sub-flow block diagram of the robot control method using the graphed visual perception features.

[0064] Figure 4 The third sub-flow block diagram of the robot control method using the graphed visual perception features.

[0065] Figure 5 The composition structure block diagram of the robot control system using the graphed visual perception features. DETAILED DESCRIPTION

[0066] In order to make the technical problems to be solved by the present application, the technical solutions and the beneficial effects more clearly understood, the following will further describe the present application in combination with the drawings and the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0067] Figure 1For the flowchart of the robot control method using the mapping visual perception feature, in the embodiment of the application, a robot control method using a mapping visual perception feature, the method comprises:

[0068] Step S100: when receiving a map generation instruction, acquiring images based on an RGB-D camera; the images include an RGB image matched in a time domain and a depth image;

[0069] When receiving a map generation instruction, images are acquired based on an RGB-D camera, the RGB-D camera is installed on a robot or a smart device and is used to acquire an image stream, wherein an RGB image is a color image, a D image is a depth image, and one RGB image and one D image are a set of inputs; it is worth mentioning that the acquired images are generally image sequences rather than single images, because the volatility of single images is too large and the analysis process is not universal, but in order to simplify the description process, the processing process of only one image is described in the technical solution of the application, in actual application, the processing process of each image needs to be applied multiple times, all images are processed, and finally the processing results are comprehensively processed to determine a final processing result, that is, a control instruction.

[0070] Step S200: target recognition is performed on the images, a target frame is located, subject recognition is performed on the content of the target frame, and subject information is determined; the subject information includes a subject name and a subject position;

[0071] The perception objects in the acquired images are normalized and expressed, redundant information is removed, and similar targets are integrated to generate a compact "object representation set", which provides clear node definition for subsequent map modeling, and the specific operation process includes:

[0072] 1. A depth detection model (such as YOLOv5, YOLOv7, YOLO-NAS, DETR or FasterR-CNN) is used to detect all targets on each frame of image.

[0073] 2. Non-maximum suppression (NMS) is applied to remove redundant detection frames that are too overlapped in the detection process.

[0074] 3. According to the position and semantic category of the target, DBSCAN (density-based clustering algorithm) is used to cluster the target, and repeated expression is eliminated to make the data more compact.

[0075] 4. Depth semantic features are extracted for each target: the image area is cropped using the target frame, and the image area is encoded into a feature vector through a feature encoding network (such as ResNet18+Transformer encoder) for subsequent processing.

[0076] In a word, the step S200 is used for locating the object in the image and identifying the object to obtain the subject name and the subject position, wherein the subject name is a word vector, and the subject position is determined by the pixel position and the depth of the image.

[0077] Step S300: constructing a node corresponding to the robot as a reference node, creating a graph containing the reference node according to the subject information, and synchronously calibrating the attributes of each node in the graph; the graph includes a spatial graph and a semantic graph;

[0078] After obtaining the subject information, the graph is created in combination with the information of the robot itself, the graph is a graph structure including nodes and edges, each node corresponds to a subject, and the node corresponding to the robot is called a reference node; at the same time, the attributes of each node in the graph also need to be calibrated, in fact, the subject name is also one of the attributes, but it is more important, so it is listed as a label.

[0079] It should be noted that the graph constructed in the technical scheme of the application includes two kinds, one is a spatial graph, and the other is a semantic graph, the spatial graph generates edges based on spatial distance, and the semantic graph generates edges based on semantic distance, the spatial graph reflects the distance of each subject in space, and the semantic graph reflects the distance of each subject in semantic space, the closer, the more likely the two subjects are to have a relationship, this process is the core of the technical scheme of the application, the object set after semantic compression is structured into a graph structure, so that the system has the modeling ability of "space + semantics + context", and provides structural cognitive ability for subsequent behavior prediction and task planning.

[0080] Step S400: receiving a demand text, identifying the demand text, and generating a control instruction according to the identification result and the graph;

[0081] In the robot control stage, the demand text such as "grabbing the water cup on the table" is received, the demand text is identified to identify the key words and actions, and then the control instruction is generated in combination with the graph, the control instruction is used to control the robot to complete the demand text, including reaching the specified position and completing the corresponding action, therefore, the control instruction actually includes a motion instruction and an action execution instruction.

[0082] As a preferred embodiment of the technical scheme of the application, a feasible data structure and its application mode are provided to complete the above steps, which are as follows:

[0083] I. Input image:

[0084] The system obtains an environment image stream through an RGB-D camera (such as RealSense D435) installed on a robot or intelligent device. Each frame of image contains an RGB image (color) and a D image (depth), forming a set of perception inputs.

[0085] II. Semantic Compression (Target Detection + Semantic Clustering):

[0086] 1) Purpose of Execution:

[0087] The semantic-level normalized expression of the perception objects in the original image is performed, redundant detection boxes are removed, and semantically similar targets are merged, outputting a compact "object representation set" that provides clear node definitions for graph modeling.

[0088] 2) Input Description: Input image frame: I ∈ R^{H×W×3} (RGB image); Pre-processed image: can be scaled and normalized in size; Depth map (optional): D ∈ R^{H×W} (from RGB-D camera).

[0089] 3) Core Processing Steps:

[0090] Target detection, using a lightweight or medium-sized deep detection model (one of the options):

[0091] YOLOv5 / YOLOv7 / YOLO-NAS (suitable for embedded), DETR (based on Transformer, supports semantic embedding), and Faster R-CNN (high precision, slow speed).

[0092] Redundant target filtering, applying non-maximum suppression (NMS) to remove overlapping boxes:

[0093] Spatial semantic clustering, clustering the remaining targets according to spatial location and semantic category to eliminate duplicate expressions:

[0094] Using DBSCAN (Density-Based Spatial Clustering).

[0095] Feature extraction, extracting a depth semantic feature vector for each target o_i (for subsequent graph construction):

[0096] Using the image region box_i for cropping, inputting a feature encoding network (such as ResNet18 + Transformer encoder), and outputting a feature vector.

[0097] III. Graph Construction (GNN Modeling):

[0098] 1) Execution purpose: structure the semantic compressed object set into a graph structure, so that the system has the modeling ability of "space + semantics + context", and provides structural cognitive ability for subsequent behavior prediction and task planning.

[0099] 2) Graph structure definition: construct graph G=(V, E, A): V: node set (each object is a node); E: edge set (relationship between nodes); A: attribute set (additional information of nodes and edges).

[0100] 3) Node construction method: each node v_i corresponds to semantic object o_i', containing the following attributes:

[0101]

[0102] Attribute table

[0103] 4) Edge construction method (E): edges represent semantic or spatial relationships between two nodes, and the construction rules are as follows:

[0104] Edge establishment condition: recommended parameters: T_dist=1m (actual space), T_sim=0.8 and T_depth=0.5m.

[0105] 5) Graph neural network modeling (graph feature fusion), using GAT (Graph Attention Network) structure to aggregate node features:

[0106] W: learnable linear transformation matrix;

[0107] a^T: attention score parameter;

[0108] h_i: input node feature, h_i': output fusion feature.

[0109] 6) Graph output structure, the final output structure is scene graph:

[0110] G=(V, E, A);

[0111] Each node v_i: entity object with semantic context within the graph;

[0112] Each edge e_ij: spatial / semantic relationship between entities;

[0113] Can be used in subsequent modules: behavior prediction, path planning, semantic question answering, etc.

[0114] Four, cognitive decision (task recognition + path planning):

[0115] Execution method: input the graph G into the decision engine, identify the task intent and plan the action path or execute the logic.

[0116] Task recognition: use subgraph matching algorithm (such as VF2 or Graph Matching Network) to identify target scene structure; task matching such as: "object A belongs to region B", "person close to table C", etc.

[0117] Behavior planning algorithm: path type task: use A* or Dijkstra shortest path algorithm;

[0118] Multi-step task sequence: use reinforcement learning (DQN, DDPG, etc.).

[0119] Five, control output (action execution):

[0120] Execution mode:

[0121] Encode the above decision results into control instructions and send them to the robot execution unit (such as mobile chassis, robot arm controller).

[0122] Communication method: ROS Topic or Service, serial control or WebSocket / API interface.

[0123] Figure 2 The first sub-flow block diagram of the robot control method using the graph-based visual perception feature, the steps of the target recognition, positioning the target frame, and subject recognition of the content of the target frame to determine the subject information include:

[0124] Step S201: frame the object in the image based on the target detection algorithm, and obtain the object frame;

[0125] Step S202: identify the object and convert the identification result into text as the subject name;

[0126] Step S203: obtain the depth of each pixel point in the object frame, and determine the object position by combining the depth and the image coordinates of the pixel point, as the subject position;

[0127] Step S204: count all subject names and subject positions in the same image to obtain the subject information corresponding to the image at the moment.

[0128] The above content limits the identification process of the image, and the object in the image is framed based on a target detection algorithm to obtain an object frame. There are many target detection algorithms, such as a deep detection model, including YOLOv5, YOLOv7, YOLO-NAS, DETR or Faster R-CNN. After locating the target, the target is framed to obtain the object frame. Then, the object is identified, and the existing object identification algorithm is applied. The object identification result is converted into text as the subject name. Then, the depth of each pixel point in the object frame is obtained, and the depth and the image coordinates of the pixel point are combined to determine the object position as the subject position. Finally, all subject names and their subject positions in the same image are counted to obtain the subject information of the image at the moment.

[0129] Figure 3 The second sub-flow chart of the robot control method using the graph-based visual perception feature is provided, the node corresponding to the robot is constructed as a reference node, and the graph containing the reference node is created according to the subject information. The steps of synchronously calibrating the attributes of each node in the graph include:

[0130] Step S301: querying the robot position, creating a node according to the robot position as a reference node;

[0131] Step S302: creating a subject node according to the subject position, and inserting the subject name as a label into the subject node;

[0132] Step S303: determining the semantic distance between the subject nodes according to the subject name of the subject node, generating edges according to the semantic distance, and obtaining a semantic graph;

[0133] Step S304: selecting the subject nodes with a semantic distance less than a preset distance threshold, calculating the spatial distance thereof, synchronously calculating the spatial distance between the reference node and each subject node, generating edges according to the spatial distance, and obtaining a spatial graph;

[0134] Step S305: inputting the subject name into a trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0135] In one example of the technical scheme of the present application, the robot position is queried, a node is created according to the robot position as a reference node, and a subject node is created according to the subject position. The subject name is inserted into the subject node as a label. On this basis, the semantic distance between the subject nodes is determined according to the subject name of the subject node, edges are generated according to the semantic distance, and a semantic graph is obtained. The semantic distance can convert the subject name into a word vector, and then the distance between the word vectors is taken as the semantic distance. Edges are generated according to the semantic distance, and a semantic graph is obtained.

[0136] After the semantic graph is generated, main body nodes with a semantic distance less than a preset distance threshold are selected, and the smaller the semantic distance, the more likely the two main body nodes appear in the same demand text, at this time, the spatial distance between them is obtained, which also means that not all main body nodes need to obtain the spatial distance; in addition, for the robot, it needs to calculate the spatial distance between each main body node, and thus, an edge is generated according to the calculated spatial distance to obtain a spatial graph.

[0137] Finally, the main body name is input into the trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value, in simple terms, the attribute analysis model is a database with data retrieval function.

[0138] As a preferred embodiment of the technical scheme of the application, the step of constructing a node corresponding to the robot as a reference node, creating a graph containing the reference node according to the main body information, and synchronously calibrating the attributes of each node in the graph further comprises:

[0139] Comparing the spatial graphs of adjacent time points to determine the position change vector of each object;

[0140] Inserting the position change vector as a dynamic attribute into the node corresponding to the object.

[0141] In an example of the technical scheme of the application, the content of the attribute is expanded, the spatial graphs of adjacent time points are compared to determine the position change vector of each object, the position change vector represents the speed in this period of time, and the position change vector is inserted as a dynamic attribute into the node corresponding to the object, at this time, the attribute contains a parameter reflecting the motion state of the object.

[0142] Figure 4 For the third sub-flow block diagram of the robot control method using the graph-based visual perception feature, the step of receiving a demand text, identifying the demand text, and generating a control instruction according to the identification result and the graph comprises:

[0143] Step S401: receiving a demand text, identifying the demand text, and extracting entities and actions;

[0144] Step S402: comparing the entities with the main body names of each node in the graph to match target nodes;

[0145] Step S403: querying the attributes of the target nodes and determining the execution state of the actions according to the attributes; the execution state includes executable and non-executable;

[0146] Step S404: selecting target nodes with an executable execution state, querying the main body positions corresponding to the target nodes, and generating motion instructions;

[0147] Step S405: When the robot reaches the main body position, an execution instruction is generated based on the action;

[0148] Step S406: When the execution is completed, a map update instruction is generated.

[0149] In an example of the technical solution of the present invention, the generation process of the control instruction is explained, the demand text is received, the demand text is recognized, and the entity and action are extracted. The entity is also a text, and its value space is the same as the value space of the subject name. The entity is compared with the subject name of each node in the graph, the target node is matched, the attribute of the target node is queried, and the execution status of the action is determined according to the attribute. For example, the attribute of the object is a fixed object. When the execution status is grasping, the execution status is not executable. The target node with an executable execution status is selected, the subject position corresponding to the target node is queried, and a motion instruction is generated. This process can be assisted by the A* or Dijkstra shortest path algorithm. The focus is on generating motion instructions for moving along the path. When the robot reaches the subject position, an execution instruction is generated based on the action. When the execution is completed, a graph update instruction is generated.

[0150] As for the atlas update instruction, it is the same as the atlas generation instruction received in step S100. Therefore, the technical solution of the present invention is in a cyclic execution state, and the control process of the robot itself is a cyclic state.

[0151] As a preferred embodiment of the technical solution of the present invention, the method further includes:

[0152] Read the text recognition results of the required text, compare the text recognition results, and calculate the text similarity;

[0153] Select the required text whose text similarity reaches the preset threshold and query its corresponding graph and motion instructions;

[0154] The neural network model is trained by using the demand text as a feature and the motion instruction as a label; wherein the motion instruction also includes the motion instruction received by the information receiving port;

[0155] The application frequency of the neural network model is adjusted according to the error rate of the neural network model.

[0156] In an example of the technical solution of the present application, a more simplified control instruction generation process is provided, in actual application, when the robot is used to work, the work content is almost repetitive, such as material grabbing and distribution in a workshop, which needs to be repeated, and the robot only performs such work for a long time, in addition, the number of work types is also small, which means that the work content of the robot is limited to a small range, on this basis, the text recognition result of the demand text is read, the text recognition result is compared, the text similarity is calculated, the demand text whose text similarity reaches the preset threshold is selected, the corresponding graph and motion instruction of the demand text is queried, thereby, the graph and motion instruction of the same demand (or very similar demand) are classified into a category, the graph and motion instruction of these demands are counted, the demand text is taken as a feature, and the motion instruction is taken as a label, and a neural network model is trained, at this time, each demand can obtain a neural network model, in actual application, the neural network model is queried according to the demand text, and the control instruction can be directly generated according to the actually constructed graph, because the sample value of the graph-control instruction is limited.

[0157] In addition, in the application process, the error rate of the neural network model also needs to be calculated in real time, and the application frequency of the neural network model is adjusted according to the error rate, the application frequency is how many times in a time period, and the application frequency is inversely proportional to the error rate.

[0158] Figure 5 For the composition structure block diagram of the robot control system adopting the graph-based visual perception feature, in the embodiment of the present application, a robot control system adopting a graph-based visual perception feature, the system 10 comprises:

[0159] An image acquisition module 11 is configured to acquire images based on an RGB-D camera when receiving a graph generation instruction; the images comprise an RGB image and a depth image after time domain matching;

[0160] A main body information determination module 12 is configured to perform target recognition on the images, locate a target frame, perform main body recognition on the content of the target frame, and determine main body information; the main body information comprises a main body name and a main body position;

[0161] A graph creation module 13 is configured to construct a node corresponding to the robot as a reference node, create a graph containing the reference node according to the main body information, and synchronize the attributes of each node in the graph; the graph comprises a spatial graph and a semantic graph;

[0162] A control instruction generation module 14 is configured to receive a demand text, recognize the demand text, and generate a control instruction according to the recognition result and the graph.

[0163] Further, the subject information determination module 12 comprises:

[0164] An object frame identification unit is configured to frame an object in an image based on a target detection algorithm to obtain an object frame.

[0165] A subject name generation unit is configured to identify the object and convert the identification result into text as a subject name.

[0166] A subject position acquisition unit is configured to acquire the depth of each pixel point in the object frame and determine the object position as a subject position by combining the depth and the image coordinates of the pixel point.

[0167] An information statistics unit is configured to count all subject names and their subject positions in the same image to obtain the subject information of the image at the moment.

[0168] Specifically, the graph creation module 13 comprises:

[0169] A reference node creation unit is configured to query the robot position and create a node as a reference node according to the robot position.

[0170] A subject node insertion unit is configured to create a subject node according to the subject position and insert the subject name as a label into the subject node.

[0171] A semantic graph generation unit is configured to determine the semantic distance between subject nodes according to the subject names of the subject nodes, generate edges according to the semantic distance, and obtain a semantic graph.

[0172] A spatial graph generation unit is configured to select subject nodes with a semantic distance less than a preset distance threshold, calculate the spatial distance thereof, synchronously calculate the spatial distance between the reference node and each subject node, generate edges according to the spatial distance, and obtain a spatial graph.

[0173] An attribute calibration unit is configured to input the subject name into a trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

[0174] Further, the control instruction generation module 14 comprises:

[0175] A text recognition unit is configured to receive a demand text, recognize the demand text, and extract entities and actions.

[0176] A node matching unit is configured to compare the entities with the subject names of each node in the graph to match a target node.

[0177] An execution state determination unit is configured to query the attributes of the target node and determine the execution state of the action according to the attributes; the execution state comprises executable and non-executable.

[0178] a motion instruction generation unit configured to select a target node with an executable execution state, query a subject position corresponding to the target node, and generate a motion instruction;

[0179] an execution instruction generation unit configured to generate an execution instruction based on the action when the robot reaches the subject position;

[0180] an update instruction generation unit configured to generate a graph update instruction when the execution is completed.

[0181] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot control method using graphical visual perception features, characterized in that: The method comprises: When receiving the atlas generation instruction, the system collects images based on the RGB-D camera; the images include the RGB images and depth images after time domain matching; Performing target recognition on the image, locating the target frame, performing subject recognition on the content of the target frame, and determining subject information; the subject information includes the subject name and subject position; Construct a node corresponding to the robot as a reference node, create a graph containing the reference node based on the subject information, and simultaneously calibrate the attributes of each node in the graph; the graph includes a spatial graph and a semantic graph; Receive the demand text, recognize the demand text, and generate control instructions based on the recognition result and the map.

2. The robot control method using graphical visual perception features according to claim 1 is characterized in that: The steps of performing target recognition on the image, locating the target frame, performing subject recognition on the content of the target frame, and determining subject information include: Select the object in the image based on the target detection algorithm to obtain the object frame; Identify the object and convert the recognition result into text as the subject name; Get the depth of each pixel in the object frame, and combine the depth and the image coordinates of the pixel to determine the object position as the subject position; Count all subject names and subject positions in the same image to obtain the subject information at the corresponding moment of the image.

3. The robot control method using graphical visual perception features according to claim 1 is characterized in that: The steps of constructing a node corresponding to the robot as a reference node, creating a graph including the reference node according to the subject information, and synchronously calibrating the attributes of each node in the graph include: Query the robot's position and create a node based on the robot's position as a reference node; Create a subject node according to the subject position and insert the subject name as a label into the subject node; Determine the semantic distance between the subject nodes according to the subject name of the subject node, generate edges according to the semantic distance, and obtain a semantic graph; Select the subject nodes whose semantic distance is less than the preset distance threshold, calculate their spatial distance, and simultaneously calculate the spatial distance between the benchmark node and each subject node. Generate edges based on the spatial distance to obtain a spatial map. The subject name is input into the trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attributes of each text are pre-stored values.

4. The robot control method using graphical visual perception features according to claim 1 is characterized in that: The steps of constructing a node corresponding to the robot as a reference node, creating a graph including the reference node according to the subject information, and synchronously calibrating the attributes of each node in the graph further include: Compare the spatial maps at adjacent moments to determine the position change vector of each object; Use the position change vector as a dynamic attribute and insert it into the corresponding node of the object.

5. The robot control method using graphical visual perception features according to claim 1 is characterized in that: The steps of receiving a demand text, recognizing the demand text, and generating a control instruction according to the recognition result and the graph include: Receive the demand text, recognize the demand text, and extract entities and actions; Compare the entity with the subject name of each node in the graph and match the target node; Query the attributes of the target node and determine the execution status of the action according to the attributes; the execution status includes executable and non-executable; Select the target node with executable status, query the main body position corresponding to the target node, and generate motion instructions; When the robot reaches the main body position, it generates execution instructions based on the action; When the execution is completed, a graph update instruction is generated.

6. The robot control method using graphical visual perception features according to claim 1 is characterized in that: The method further comprises: Read the text recognition results of the required text, compare the text recognition results, and calculate the text similarity; Select the required text whose text similarity reaches the preset threshold and query its corresponding graph and motion instructions; The neural network model is trained by using the demand text as a feature and the motion instruction as a label; wherein the motion instruction also includes the motion instruction received by the information receiving port; The application frequency of the neural network model is adjusted according to the error rate of the neural network model.

7. A robot control system using graphical visual perception features, characterized in that: The system comprises: An image acquisition module is configured to acquire images based on an RGB-D camera upon receiving an atlas generation instruction; the images include an RGB image and a depth image after time domain matching; A subject information determination module is used to perform target recognition on the image, locate the target frame, perform subject recognition on the content of the target frame, and determine the subject information; the subject information includes the subject name and the subject position; A graph creation module is used to construct nodes corresponding to the robot as reference nodes, create a graph containing the reference nodes based on the subject information, and simultaneously calibrate the attributes of each node in the graph; the graph includes a spatial graph and a semantic graph; The control instruction generation module is used to receive the demand text, identify the demand text, and generate the control instruction according to the recognition result and the map.

8. The robot control system using graphical visual perception features according to claim 7, characterized in that: The subject information determination module includes: An object frame recognition unit is used to select an object in an image based on a target detection algorithm to obtain an object frame; A subject name generation unit is used to identify the object and convert the recognition result into text as the subject name; A subject position acquisition unit, configured to acquire the depth of each pixel in the object frame, and determine the object position by combining the depth and the image coordinates of the pixel as the subject position; The information statistics unit is used to count the names and positions of all subjects in the same image to obtain the subject information at the corresponding moment of the image.

9. The robot control system using graphical visual perception features according to claim 7, characterized in that: The graph creation module includes: A reference node creation unit is used to query the robot position and create a node according to the robot position as a reference node; A subject node insertion unit is used to create a subject node according to the subject position and insert the subject name as a label into the subject node; A semantic graph generation unit is used to determine the semantic distance between subject nodes according to the subject names of the subject nodes, generate edges according to the semantic distance, and obtain a semantic graph; A spatial graph generation unit is used to select subject nodes whose semantic distance is less than a preset distance threshold, calculate their spatial distance, simultaneously calculate the spatial distance between the reference node and each subject node, generate edges based on the spatial distance, and obtain a spatial graph; The attribute calibration unit is used to input the subject name into the trained attribute analysis model to calibrate the attributes of each node in the image; wherein the attribute analysis model is a data query model, and the attribute of each text is a pre-stored value.

10. The robot control system using graphical visual perception features according to claim 7, characterized in that: The control instruction generation module includes: A text recognition unit is used to receive the demand text, recognize the demand text, and extract entities and actions; The node matching unit is used to compare the entity with the subject name of each node in the graph and match the target node; An execution status determination unit, configured to query the attributes of a target node and determine the execution status of an action according to the attributes; the execution status includes executable and non-executable; A motion instruction generating unit is used to select a target node with an executable execution state, query the main body position corresponding to the target node, and generate a motion instruction; an execution instruction generating unit, configured to generate an execution instruction based on the action when the robot reaches the main body position; The update instruction generation unit is used to generate a graph update instruction when the execution is completed.

Images