A robotic question answering method for item search

Abstract

The present application belongs to the technical field of image perception and language processing, and particularly relates to a robot question-answering method for article search. The method first inputs an image to be used for constructing a scene atlas after the image is labeled with an object bounding box and a category label by a residual-feature classification network, and the subsequent input image sequence is used for updating the scene atlas; a feature vector is output after a long short-term memory network is used to process an input question, and a keyword is obtained after the feature vector is processed by a recurrent convolution classification network; and a breadth-first search is performed in the scene atlas to obtain a question answer and output the question answer. The method breaks through the defects in traditional question-answering systems, such as over-reliance on pre-provided expert data and lack of physical interaction process in a real environment, realizes autonomous operation of a robot, active environment interaction, semantic analysis, improves the interactivity and intelligence of the robot question-answering system, and has great application prospects in various scenes including express delivery storage, shopping malls, factories and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image perception and language processing, and particularly relates to a robot question-answering method for object search. BACKGROUND

[0002] In recent years, with the continuous development of robot technology, people have higher requirements for robots, especially service robots, and gradually require robots to complete certain tasks under human natural language instructions. However, in actual application scenarios, such as home environment, there are often many objects, and some objects are even hidden in positions that cannot obtain the field of view, which makes it difficult for robots to complete simple tasks by simply obtaining the field of view. Therefore, robots need to introduce more execution methods to complete tasks according to human language instructions in diversified, complex and large-scale environments.

[0003] In the current question-answering system that applies artificial intelligence technology the most, the main means are the classic question-answering of analyzing language instructions and searching a preset database and the visual question-answering of answering questions by visual image analysis using the current field of view, and there is no method of using robot operation to assist question-answering. Simple expert question-answering systems can only realize the processing and feedback of language information, and service robots currently applied in hotels, restaurants and other public places can only further obtain information and answer questions by means of navigation, movement and the like. When facing scenes that need to process a large number of various objects or indoor furnishings are relatively complex, such as supermarkets, auto repair shops and the like, robots cannot realize active operation and exploration of the environment according to language instructions to answer questions.

[0004] Embodied intelligence is a new artificial intelligence implementation method in the field of artificial intelligence in recent years from the perspective of behavior. In embodied intelligence, researchers use the method of "cognition is calculation", that is, the agent constructs its own knowledge structure or cognitive model by interacting with the environment without generating too many prior knowledge rules. However, due to the difficulty of interacting with the real environment, the current mainstream embodied intelligence research is mainly limited to simulation environments, and there are few researches on real robots. The few researches applied in real environments are limited to simple interactive forms such as movement navigation, and lack of interactive forms that can perform movement or object shape changing operations, which greatly limits the practical application of embodied intelligence. At present, embodied intelligence also lacks practical application in the field of robots. SUMMARY

[0005] The purpose of the present application is to propose a robot question-answering method for object search, which breaks through the defects that the traditional question-answering system too much relies on pre-given expert data and lacks physical interaction process in real environment, and applies it to actual robots to realize the question-answering process based on operation interaction.

[0006] The robot question answering method for object search provided by the application firstly inputs an image to label the object boundary box and the category label through a residual-feature classification network, and then uses the image to construct a scene graph, and the subsequent input image sequence is used to update the scene graph. After the input question is processed through a long short-term memory network, a feature vector is output, and after the feature vector is processed through a recurrent convolution classification network, a keyword is obtained and a breadth-first search is performed in the scene graph to obtain a question answer and output.

[0007] The robot question answering method for object search provided by the application has the following advantages:

[0008] 1. The robot question answering method for object search provided by the application is based on a real environment, and a binocular camera RGB-D visual field collection and processing module and a corresponding 6-degree-of-freedom mechanical arm operation system are designed, so that the robot can learn from the existing environment interaction without over-reliance on the pre-provided expert database in the actual application of the robot question answering for a specific scene. The environment with various object types, complex furnishings and inconvenient pre-constructed database can be applied.

[0009] 2. The method provided by the application simultaneously uses multiple image frames and the corresponding scene graph technology that can be dynamically updated, and solves the problem that the traditional visual question answering system cannot handle dynamic processes with operations. Therefore, the problems existing in the robot question answering system of the prior art, such as the limited information that can be obtained by the visual method, especially the method of analyzing and judging based on static images, and the inability to analyze problems containing dynamic processes, are solved.

[0010] 3. In the method provided by the application, the embodiment intelligence design idea is adopted to realize the robot autonomous operation, active environment interaction and semantic analysis, and the interactivity and intelligence of the robot question answering system are improved, and the method has great application prospects in various scenes including express delivery storage, shopping malls, factories and the like. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flow chart of the robot question answering method in the object search process provided by the application.

[0012] Figure 2 is a schematic diagram of the residual-feature hierarchical network architecture (ResNet-FPN) involved in the method provided by the application. DETAILED DESCRIPTION

[0013] The robot question-answering method for article search provided by the application firstly inputs an image to label an object bounding box and a category label through a residual-feature classification network, and then uses the image to construct a scene graph, and a subsequent input image sequence is used to update the scene graph. After a long short-term memory network processes an input question, a feature vector is output, and after the feature vector is processed through a recurrent convolution classification network, a keyword is obtained and a breadth-first search is performed in the scene graph to obtain a question answer and output the question answer.

[0014] The robot question-answering method for article search described above has a flowchart as shown in Figure 1 The specific process includes the following steps:

[0015] (1) An RGB depth image of a plurality of articles is shot, and the plurality of depth images are respectively labeled with article categories;

[0016] (2) RGB depth images of a plurality of scenes in a search area of a robot are successively shot and recorded as scene images, the plurality of scene images are respectively scaled to three-dimensional images with uniform sizes, the size of the three-dimensional image is w*w*h, in an embodiment of the application, the size of the three-dimensional image is 224*224*3, each scene image respectively contains three layers, the size of each layer is w*w, and each layer in the scene image with uniform sizes is normalized by using the following formula:

[0017]

[0018] In the above formula, i is the number of a pixel point in a layer, represents the value of the pixel point i in the normalized scene image, x i represents the value of the pixel point i in the scene image with uniform sizes, represents the average value of pixel point values of all pixel points in the layer, std x represents the standard deviation of pixel point values of all pixel points in the layer;

[0019] (3) A residual-feature hierarchical network (ResNet-FPN) convolutional neural network is used to obtain a scene image feature of the search area, and the process includes the following steps:

[0020] (3-1) The normalized scene image of step (2) is input into the residual-feature hierarchical network (ResNet-FPN) convolutional neural network as shown in Figure 2 The convolutional neural network outputs a scene image feature of the search area, and the scene image feature of the search area is a plurality of feature matrices with different levels; in an embodiment of the application, there are four feature matrices P1, P2, P3 and P4, as shown in Figure 2The pre-processing simulation system and the real camera sensor collect object pictures with category labels of 256*256*3 size, and then the ResNet part with five convolution layers is imported. In each convolution layer of the part, a 7*7 size convolution kernel with a step of 2 is used to extract features. After each convolution layer, a 2*2 size pooling kernel is used for maximum pooling in the pooling layer, and a ReLU function is used as a nonlinear activation function in the activation layer. Under this architecture, the input picture is output with different levels of features with the original size 0.5*0.5 after each convolution layer, and then the features are added after a 1*1 convolution to reduce the channel number, and finally the features are output after a 3*3 convolution.

[0021] (3-2) Repeat step (3-1) to traverse all normalized scene images of step (2) to obtain scene image feature matrices of multiple search regions;

[0022] (4) Construct an item search model, train the item search model, that is, input the scene image feature matrix of step (3) into the subsequent part of the residual-feature hierarchical network (ResNet-FPN) convolutional neural network of step (3) to perform fusion feature regression, obtain the bounding box of each item in the scene image, perform region of interest (ROI) pooling and alignment on the region inside the item bounding box, extract the region feature inside the item bounding box, output the item category, and obtain an initial item search model;

[0023] (5) Traverse all scene image feature matrices in step (3) in turn, repeat step (4), and obtain a final item search model;

[0024] (6) Use a reinforcement learning method to generate multiple robot operation actions, record the scene images at different sampling moments in the operation process, or sample the operation process of the robot in a real environment, and record the scene images at different sampling moments in the operation process. Use the method of step (2) to scale the above scene images to w*w*h images with the same size as step (2), and input the multiple frames of scaled images I1, I2, I3… as image sequences and scene search problems into the robot operation interactive question and answer system;

[0025] (7) Input the first image in the image sequence of step (6) into the item search model of step (5), obtain the given bounding box of the item and the category of the item from the output result of the item search model, add the obtained category of the item as a node to an initialized scene graph, and judge the relative position relationship between the items according to the geometric relationship between the item bounding boxes, and add the judgment result as an adjacent edge to the above initialized scene graph to obtain a scene graph.

[0026] (8) traversing all images in the image sequence of step (6), repeating step (7), updating the scene graph, the updating process is: finding the largest bounding box of the object in the center region of the current frame scene image, aligning the object in the current frame with the object in the previous frame based on the object, identifying and determining the position relationship of the object in the current frame in the scene graph, according to the identification and determination result, starting from the node where the object is located in the previous frame scene graph, updating the scene graph to obtain a plurality of scene graphs;

[0027] (9) constructing a natural language processing module based on long short-term memory network (LSTM), inputting the object search question into the language encoding network of the natural language processing module to obtain the encoding and keywords of the search question, the encoding form is a feature vector, the language encoding network contains three layers of long short-term memory network, the number of neurons in each layer is 128, the input question is converted into a 128-dimensional feature vector according to the vocabulary table constructed according to the word frequency in step (7) and the keywords are extracted, the keywords contain the object category and the position relationship. Input the feature vector into the classification recurrent neural network in the natural language processing module to obtain the type of the object search question; the classification recurrent neural network contains three layers, the first layer is long short-term memory network (LSTM), the number of neurons is 64, the second layer is linear transformation layer, and the last layer is softmax full connection layer. The classification value is obtained by classifying the processed feature vector to determine whether the question belongs to "whether question" or "counting question";

[0028] (10) according to the encoding feature vector, search question type and keywords obtained in step (9), searching the multiple scene graphs corresponding to the multiple scene images obtained in step (8), that is, according to the keywords, traversing all nodes in the multiple scene graphs to obtain the node position of the searched object in each frame of the scene graph; using breadth first method (BFS), searching the adjacent nodes of the node where the searched object is located according to the search question type and keywords, determining the multiple adjacent nodes and the adjacent edges corresponding to the adjacent nodes that meet the search condition, selecting a frame scene graph with the largest number of adjacent edges from the multiple frame scene graphs in the search order, searching the adjacent nodes of the node where the searched object is located and the adjacent edges corresponding to the adjacent nodes in the scene graph to obtain the answer to the object search question and output. For example, for "whether question", the answer is "yes" or "no", and for "counting question", the answer is "0", "1", "2", etc.

[0029] In step (7) of the above-mentioned robot question and answer method for object search, the relative position relationship between the objects is determined according to the geometric relationship between the object bounding boxes, and the specific process is as follows:

[0030] (1) For the object O1 and the object O2 with the boundary box, the coverage rate r and the coincidence degree l are calculated respectively:

[0031]

[0032] Wherein, S1, S2 represent the area of the boundary box, S overlap represents the coincidence area of the boundary box;

[0033]

[0034] Wherein, L1, L2 represent the diagonal length of the boundary box, d center represents the center distance of the boundary box;

[0035] (2) The threshold value x of the coverage rate r and the threshold value y of the coincidence degree l are set respectively, in one embodiment of the present application, x = 0.5, y = 1, the coverage rate r and the coincidence degree l calculated in step (1) are judged:

[0036] If r >= x, it means that the coverage rate of the object O1 and the object O2 is high, then the relative position relationship of the object O1 and the object O2 is determined as “up / down”;

[0037] If r < x and l < y, it means that the area coincidence rate of the object O1 and the object O2 is not high, but the centers are close enough, then the relative position relationship of the object O1 and the object O2 is determined as “up / down”;

[0038] If r < x and x <= l < y, it means that the coincidence rate is not high, but the position is close, then the relative position relationship of the object O1 and the object O2 is determined as “close”;

[0039] If r < x and l >= y, it means that the two objects are far away from each other, then the relative position relationship of the object O0 and the object O2 is determined as no special relative position relationship.

[0040] The specific implementation process of the robot question and answer method for object search of the present application includes:

[0041] In the robot system simulation environment, the object image is sampled or the object RGB-D image is collected by shooting in the real environment and the corresponding object category is labeled to construct the object recognition data for pre-training the object category recognition model. Then, the model is used as the object detector to construct an operation interactive question and answer system which can process multiple input image frames and corresponding questions and give answers, wherein the image processing part can detect the object category in each picture and label the boundary box to determine the relative position relationship between objects, and can align each object to the corresponding position in the previous frame to update the corresponding nodes and adjacent edges in the corresponding scene graph; the language instruction processing module can encode the input question into a word vector form and preliminarily determine the question type. The question and answer analysis module splices the above image information and question code to obtain a feature fusion vector, and uses a convolution network to process the feature fusion vector to find the key node in the scene graph corresponding to the question and retrieve the answer. For the input yes or no question, the output is "yes, no", and for the input counting question, the output is "1, 2, 3...".

[0042] The following describes an example of the system of the method of the present application in conjunction with the accompanying drawings:

[0043] (1) The scene RGB-D image taken by the binocular depth camera suspended on the support above the operation area of the robot simulation system or in the actual system is called a scene image. In the construction process of the data set, the scene image is generally reduced to an image with a size of 224*224*3, and the scene image contains three layers, and the size of each layer is 224*224. Then, the original image can be normalized by using the following formula:

[0044]

[0045] In the above formula, represents the value of the pixel point in the normalized environment image, x i represents the value of the pixel point in the original environment image, represents the average value of all pixel points in the layer, std x represents the standard deviation of all pixel points in the layer.

[0046] (2) Construct a ResNet-FPN architecture of convolutional neural network for training object detector, specifically including the following steps: preprocessing the object pictures with class labels collected by simulation system and real camera sensor to 256*256*3 size, then importing the ResNet part with five convolutional layers, in each convolutional layer, a 7*7 size convolution kernel with a step of 2 is used to extract features, after each convolutional layer, a pooling layer with a 2*2 size pooling kernel is used for maximum pooling, and an activation layer with ReLU function as the nonlinear activation function is used, under this architecture, the input picture is output with different levels of features with the original size 0.5*0.5 after each convolutional layer, then after reducing the channel number by 1*1 convolution, the features are added again and output the final features by 3*3 convolution.

[0047] (3) Use the convolutional neural network of step (2) above to process the input image to extract multi-level feature matrices P1, P2, P3, P4 with sizes of 128, 64, 32, and 16 respectively, and then input them into the subsequent Fast-RCNN network for feature fusion and regression to obtain the bounding box of the target object, and further perform ROI pooling and alignment on the region inside the bounding box to extract features and analyze the output object label category, thereby completing the construction of the object detector.

[0048] (4) In the simulation system or real environment, a series of operations are generated by the reinforcement learning method for a given problem, and scene images at different time points during the operation process are sampled and scaled to 224*224*3 size. The multi-frame images I1, I2, I3… are input as image sequences into the operation interactive question and answer system for the given problem.

[0049] (5) Based on the image sequence input in (4), a knowledge graph is constructed, first, the object detector obtained in (3) is used to identify the object in the input image sequence and give the bounding box and determine the category, the obtained object category label is added to the scene graph as a node, and the relative position relationship between the objects is determined as the adjacent edge added to the graph according to the geometric relationship between the object bounding boxes, and the construction of the scene graph is completed. The judgment criteria for the relative position relationship of the objects are as follows:

[0050] For the objects O1, O2 with bounding boxes, two indicators are calculated: coverage rate r and overlap degree l:

[0051]

[0052] Where S1, S2 represent the areas of the two bounding boxes, S overlap represents the overlapping area of the two bounding boxes

[0053]

[0054] wherein L1, L2 represent the length of the diagonal of the bounding box of both, d center represents the distance of the center of both bounding boxes.

[0055] If r > 0.5 indicates that the coverage of both is high, the relationship is determined as "above / below".

[0056] If r < 0.5 and l < 0.5 indicates that the area overlap rate of both is not high but the centers are close enough to each other, the relationship is also determined as "above / below".

[0057] If r < 0.5 and 0.5 < l < 1 indicates that the overlap rate of both is not high but the positions are relatively close, the relationship is determined as "adjacent".

[0058] If r < 0.5 and l > 1 indicates that both are far away from each other, it is determined that there is no special relative position relationship. For the first frame of scene image, the above method constructs an initial scene graph and then aligns the corresponding positions in the last frame based on the object in the center area of the maximum bounding box of the image after each frame input, re-identifies and determines the relative position relationship and updates the scene graph.

[0059] (6) A natural language processing module is constructed to process the input question. First, a language encoding network is constructed to encode the input question into a vector. The language encoding network includes three layers of long short-term memory networks, and the number of neurons in each network is 128. According to the vocabulary table constructed according to the pre-statistics of the input question word frequency, the input question is converted into a 128-dimensional word vector and the key words are extracted. The obtained feature vector is then processed by a classification recurrent neural network to determine the question type. The network includes three layers, the first layer is a long short-term memory network with 64 neurons, the second layer is a linear transformation layer, and the last layer is a softmax fully connected layer. After the input word vector is processed to extract features, the classification value is obtained to determine whether the question belongs to "whether the question" or "counting question".

[0060] (7) According to the key word, the key node is searched to output the answer to the question:

[0061] According to the word feature vector, question type, and key word obtained in the foregoing (6), the scene graph obtained in (5) is searched. First, the key object position is located by traversing the graph nodes frame by frame according to the key word, and then the breadth-first method is used to search the object nodes with special relative position relationship and determine the frame with the maximum number of nodes and adjacent edges as the key frame to search the adjacent nodes and corresponding adjacent edges to output the answer to the question.

[0062] In the robot system simulation environment, the object image is sampled or the object RGB-D image is collected by shooting in the real environment, and the object category is labeled to construct the object recognition data for pre-training the object category recognition model. Then, the model is used as the object detector to construct an operation interactive question and answer system which can process multiple input image frames and corresponding questions and answer at the same time. The image processing part can detect the object category in each picture and label the boundary box to determine the relative position relationship between objects, and can align each object to the corresponding position in the previous frame to update the corresponding node and adjacent edge in the corresponding scene graph. The language instruction processing module can encode the input question into a word vector form and preliminarily determine the question type. The question and answer analysis module splices the above image information and question code to obtain a feature fusion vector, and uses a convolutional network to process the feature fusion vector to find the key node in the scene graph corresponding to the question and retrieve the answer. For the input yes or no question, the output is "yes, no", and for the input counting question, the output is "1, 2, 3...".

Claims

1. A robot question-answering method for item search, characterized in that, The specific process of the method comprises the following steps: (1) Take RGB depth images of a plurality of objects, and respectively label the object categories of the plurality of depth images; (2) Take RGB depth images of a plurality of scenes in the search area of the robot in sequence, and record them as scene images, and normalize each layer in the scene images; (3) Use the convolutional neural network in the residual-feature hierarchical network to obtain the scene image features of the search area; (4) Construct an object search model and train the object search model; (5) Traverse all scene image feature matrices in step (3) in sequence, repeat step (4), and obtain a final object search model; (6) Use reinforcement learning to generate a plurality of robot operation actions, record the scene images at different sampling times during the operation process, or sample the operation process of the robot in a real environment, and record the scene images at different sampling times during the operation process; (7) Input the first image in the image sequence in step (6) into the object search model in step (5), obtain the object given bounding box and object category from the output result of the object search model, add the obtained object category as a node to an initialized scene graph, and judge the relative position relationship between the objects according to the geometric relationship between the object bounding boxes, and add the judgment result as an adjacent edge to the above-mentioned initialized scene graph, to obtain a scene graph; (8) Traverse all images in the image sequence in step (6), repeat step (7), update the scene graph, and the update process is: find the object with the largest bounding box in the central region of the current frame scene image, align the position of the object in the current frame with the position of the object in the previous frame based on the object, identify and determine the position relationship of the object in the current frame in the scene graph, and update the scene graph in the previous frame based on the object node as the starting point according to the identification and determination result, to obtain a plurality of scene graphs; (9) Construct a natural language processing module based on a long short-term memory network (LSTM), input the object search question into the language encoding network in the natural language processing module, obtain the encoding and keywords of the search question, the encoding form is a feature vector, input the feature vector into the classification recurrent neural network in the natural language processing module, and obtain the type of the object search question; (10) searching the multiple scene graphs corresponding to the multiple scene images obtained in step (8) according to the encoded feature vector obtained in step (9) and the search question type and the keyword, that is, traversing all nodes in the multiple scene graphs according to the keyword to obtain the node position of the searched item in each frame of the scene graph; searching the adjacent nodes of the node where the searched item is located according to the search question type and the keyword by using the breadth-first method to determine the multiple adjacent nodes of the relative position relationship meeting the search condition and the adjacent edges corresponding to the adjacent nodes, selecting a frame of the scene graph with the largest number of adjacent edges from the multiple scene graphs in the search order, searching the adjacent nodes of the node where the item is located and the adjacent edges corresponding to the adjacent nodes in the scene graph to obtain the answer to the item search question and outputting.

2. The robot question answering method for object search of claim 1, wherein In the step (7), the relative position relationship between the items is judged according to the geometric relationship between the item bounding boxes, and the specific process is as follows: (1) For the item with the bounding box labeled , calculate the coverage and the overlap , respectively: wherein, represents the area of the bounding box of both, represents the area of the bounding box of both; wherein, represents the length of the diagonal of the bounding box of both, represents the distance of the centers of the bounding boxes of both; (2) Set the coverage rate separately threshold and overlap threshold The coverage calculated in step (1) and overlap Make a judgment: like , indicating items If the coverage is high, then the item is considered... The relative positional relationship is "up / down"; If At the same time , the area coincidence rate of the objects is not high, but the centers are close enough to each other, it is determined that the relative positional relationship of the objects is "up / down"; If At the same time , indicates that the coincidence rate of both is not high, but the position is relatively close, then the relative position relationship of the article "close"; If At the same time , indicates that the two are away from each other, then the relative position relationship of the article Is not a special relative position relationship.

Images