A knowledge-enhanced multi-service robot object category recognition method and system

Abstract

This invention belongs to the field of service robot visual scene recognition technology, and provides a knowledge-enhanced method and system for object category recognition in multi-service robots. The method acquires scene images of the robot's operation, extracts visual information feature maps from these images, obtains preliminary predictions of target detection boxes and their object categories, and converts these visual information feature maps into scene information vectors. Trusted target label nodes from a pre-constructed knowledge graph are selected, and a matching vector is formed based on the converted scene information vector and target detection box information. The similarity between the matching vector and edges starting from the trusted target label nodes is calculated, and the query prediction result for the object category in the scene image is determined based on the similarity. Finally, the query prediction result for the object category in the scene image is fused with the preliminary prediction result to obtain the final object category recognition result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of visual scene recognition of service robots, and particularly relates to a knowledge-enhanced multi-service robot object category recognition method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Service robots have important significance for improving people's daily life, and have been widely used in old-age care, education and rescue, guardianship and other work. The work of service robots cannot be separated from the perception and understanding of the surrounding environment by robots, and visual signals are an important source of environmental perception by robots. Making full use of visual information helps to improve the accuracy of target classification and the accuracy and reliability of downstream tasks.

[0004] In the current widely used service robot object recognition algorithm, there are the following two problems: 1) lack of accumulation of prior knowledge: since the current visual model is mostly obtained by offline training, it cannot be dynamically updated as the robot runs; 2) the accumulated prior knowledge cannot be shared among multiple service robots. SUMMARY

[0005] In order to solve the technical problems existing in the background art, the present application provides a knowledge-enhanced multi-service robot object category recognition method and system, which uses a knowledge graph as an experience storage and experience sharing platform based on existing visual recognition algorithms, accumulates object recognition content with high confidence during robot operation into a database, and supplements the visual information graph at regular intervals to solve the problem of accumulation of prior knowledge. At the same time, the graph database for storing the graph provides services for multiple servers at the same time to realize the sharing of object recognition knowledge.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a knowledge-enhanced multi-service robot object category recognition method.

[0008] A knowledge-enhanced multi-service robot object category recognition method comprises:

[0009] Obtaining a scene picture of robot operation, extracting a visual information feature map in the scene picture, obtaining a preliminary prediction result of a target detection frame and its object category in the scene picture, and converting the visual information feature map into a scene information vector.

[0010] Based on the preliminary prediction results of the above object categories, trust target label nodes in the pre-constructed knowledge graph are selected. The similarity between the matching vector and the edge starting from the trust target label node is calculated based on the transformed scene information vector and the target detection box information. The query prediction result of the object category in the scene image is determined according to the similarity.

[0011] The query prediction results of object categories in the scene image are fused with the preliminary prediction results to obtain the final object category recognition result.

[0012] As one implementation method, the classification confidence of the preliminary prediction results of the object categories in the scene image is calculated, the object category bounding boxes with the preliminary classification confidence of the results being higher than a set threshold are saved, and the knowledge graph is updated.

[0013] As one implementation method, during the process of updating the knowledge graph, the vector value of the corresponding directed edge in the updated knowledge graph is: the weighted sum of the first vector value and the second vector value;

[0014] The first vector value is the vector value corresponding to the directed edge in the original knowledge graph, and its weight is a preset update speed adjustment factor; the second vector value is the scene information vector value corresponding to the preliminary prediction result of the object category in the scene image, and its weight is the difference between 1 and the preset update speed adjustment factor.

[0015] As one implementation method, the knowledge graph is composed of directed edges and nodes, where nodes represent the identified object label values, and the starting point of the directed edge is the object with high certainty, and the ending point is the inferred object.

[0016] As one implementation method, the final object category recognition result is the weighted sum of the query prediction result and the preliminary prediction result of the object category in the scene image.

[0017] As one implementation method, the visual information feature map is converted into a scene information vector through a fully connected layer.

[0018] In one implementation, the similarity is characterized by cosine similarity.

[0019] A second aspect of the present invention provides a knowledge-enhanced object category recognition system for multi-service robots.

[0020] A knowledge-enhanced multi-service robot object category recognition system, comprising:

[0021] The preliminary prediction module is used to acquire scene images of robot operation, extract visual information feature maps from the scene images, obtain preliminary prediction results of target detection boxes and their object categories in the scene images, and convert the visual information feature maps into scene information vectors.

[0022] The query prediction module is used to select trusted target label nodes in a pre-constructed knowledge graph based on the preliminary prediction results of the above object categories, form a matching vector based on the transformed scene information vector and target detection box information, calculate the similarity between the matching vector and the edge starting from the trusted target label node, and determine the query prediction result of the object category in the scene image based on the similarity.

[0023] The prediction fusion module is used to fuse the query prediction results of the object category in the scene image with the preliminary prediction results to obtain the final object category recognition result.

[0024] A third aspect of the present invention provides a computer-readable storage medium.

[0025] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the knowledge-enhanced multi-service robot object category recognition method as described above.

[0026] A fourth aspect of the present invention provides an electronic device.

[0027] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the knowledge-enhanced multi-service robot object category recognition method described above.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] This invention uses a knowledge graph as a platform for experience storage and sharing. It accumulates high-confidence object recognition data from robot operations into a database and periodically adds it to the visual information graph to solve the problem of prior knowledge accumulation. Simultaneously, it provides services to multiple servers through a graph interface to achieve the sharing of object recognition knowledge.

[0030] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0031] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0032] Figure 1 This is a schematic diagram of the object recognition knowledge graph structure according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the principle of knowledge-enhanced multi-service robot object category recognition in an embodiment of the present invention.

[0034] Figure 3 This is a flowchart illustrating the operation of the robot according to an embodiment of the present invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] Example 1

[0039] according to Figure 2 and Figure 3 This embodiment provides a knowledge-enhanced multi-service robot object category recognition method, which includes:

[0040] Step 1: Obtain scene images of the robot's operation, extract visual information feature maps from the scene images, obtain preliminary prediction results of target detection boxes and their object categories in the scene images, and convert the visual information feature maps into scene information vectors.

[0041] In step 1, a visual information extraction model can be used to extract visual information feature maps from the scene image.

[0042] For example, Faster R-CNN was used as a visual information extraction model and trained on the COCO 2017 dataset. This enabled the model to detect objects in a scene using visual features, extract information such as the size and position of the objects, and make a preliminary prediction of the category of the detected objects.

[0043] It is understood here that in other embodiments, other existing neural network model structures may also be used to construct the visual information extraction model.

[0044] The parameters of the visual information extraction model are fixed, and the feature maps generated during its inference process are used as input to the scene classification model. After the feature maps are unfolded, they are passed through three fully connected layers to obtain a 64-dimensional image scene information vector, which is defined as the scene information vector. The scene information vector is then passed through another fully connected layer and a softmax layer to output the classification information of the scene image. The classifiable categories include eight types of indoor scenes (bedroom, living room, kitchen, balcony, toilet, study, corridor, elevator) and non-indoor scenes.

[0045] During the training of the visual information extraction model, the model is trained on a labeled dataset. The parameters of the fully connected layer in the scene classification model are trained so that it can distinguish the current scene category.

[0046] The images of each scene in the dataset are processed using a visual information extraction model, and the resulting feature maps are input into a scene classification model to obtain scene information vectors for each image in the dataset. As shown in the following formula, the scene feature vectors are averaged by scene category, and the resulting vector is used as the scene feature vector for that category to represent that type of scene.

[0047]

[0048] In the formula, v k v represents the vector representation of the k-th scenario. ki The scene is represented by the vector corresponding to the i-th sample of the k-th class, which is represented by the trained scene classification model, n k This represents the number of samples in the dataset whose scenario is class k.

[0049] During robot operation, the scene image is first input into the visual information extraction model to obtain the possible bounding boxes and their possible classification information in the current scene image. The bounding boxes detected using only image information are ranked according to their confidence scores, and the top m bounding boxes with confidence scores higher than a threshold are selected as trusted targets. The final output of the visual information extraction model is the recognition result.

[0050] The feature map obtained from the visual information extraction model is passed through the trained network to obtain a scene information vector describing the current image and a scene classification prediction result. If the classification result indicates a non-indoor environment, the output of all detection boxes is the detection result of the visual detection model. If the current environment is an indoor scene, the current scene feature vector is saved.

[0051] Step 2: Based on the preliminary prediction results of the object category, select the trusted target label node in the pre-constructed knowledge graph, form a matching vector based on the transformed scene information vector and the target detection box information, calculate the similarity between the matching vector and the edge starting from the trusted target label node, and determine the query prediction result of the object category in the scene image according to the similarity.

[0052] Knowledge graphs consist of directed edges and nodes, such as... Figure 1 As shown, nodes represent the identified object label values, and the starting point of a directed edge is an object with high certainty, while the ending point is an object that can be inferred from that object. The value of a directed edge is a 67-dimensional feature vector, where the first 64 dimensions are the feature vector of the current scene, and the last 3 dimensions are the visual position information (s, p). x ,p y ), where s represents the ratio of the area of ​​the object to be predicted to the area of ​​the object to be determined, and p x ,p y The calculation method is shown in the following formula:

[0053] l=min{width(obj1),height(obj1)}

[0054]

[0055]

[0056] In the formula, obj1 represents the defined object, obj2 represents the object to be predicted, width(obj) and height(obj) represent the width and height of the object, respectively, and x(obj) and y(obj) represent the x-coordinate and y-coordinate of the center point of the object, respectively.

[0057] Based on the training data in the dataset, visual position information between objects in different scenarios is obtained, and then connected with the feature vectors of various scenarios. The resulting vector is the value of the edge in the knowledge graph.

[0058] Suppose there are n bounding boxes whose classification confidence is below a threshold and whose classification value is to be determined. For each bounding box, first obtain m sets of visual location information (s, p) based on m trusted bounding boxes. x ,p y ), where s represents the ratio of the area of ​​the predicted object to the area of ​​the determined object.

[0059] After obtaining m sets of visual position information, each set is concatenated with the scene information vector of the current image to obtain m sets of matching feature vectors. Therefore, a total of n*m matching feature vectors are obtained, represented as: H={h 11 ,h 12 ,h 13 ...h nm}, where n is the number of target boxes to be predicted and m is the number of trusted target boxes.

[0060] Using each set of feature vectors as a basis, the knowledge graph is queried, and n sets of matching feature vectors are traversed. The trusted target label node corresponding to the matching feature vector is found in the graph. Cosine similarity is calculated between the matching feature vector and the edge starting from that node. The similarity score is used as the category prediction result for the target detection box. Specifically, for the i-th target box, there are m matching vectors, denoted as h... i1 ,h i2 ,…,h im h ij Calculate the cosine similarity with each of the t edges of node j, obtaining t similarity values, and then use these values ​​as a vector p. ij , where t is the number of object types that can be identified. Similarly, we obtain p. i1 ,p i2 ,…,p im Calculate the average value of these values ​​to obtain p. i This represents the category probability distribution of the i-th target box obtained through knowledge graph reasoning. Performing this operation sequentially on other target boxes yields their respective category probability distributions obtained through knowledge graph reasoning.

[0061] Step 3: Fuse the query prediction results of the object categories in the scene image with the preliminary prediction results to obtain the final object category recognition results.

[0062] The obtained prediction results are fused with the visual information extraction model results according to the following formula to obtain the final target box classification value.

[0063] p label =α·p graph +(1-α)·p view

[0064] In the formula p view p represents the prediction result of the visual information extraction model for classifying the bounding box. graph The prediction result is based on the atlas used in this step. α is an adjustment factor used to set the weight of the influence of information in the atlas on the target box classification result. The value range is 0-1.0, and it needs to be manually adjusted according to the actual scenario. Generally, the value range is 0.2-0.6. label This is the predicted result of the final classification value after fusion.

[0065] During operation, multiple robots send the information of m trustworthy bounding boxes with confidence levels higher than a threshold from the preliminary results output by the visual information extraction model in the current scene to the graph backend interface. The graph backend stores the acquired information in the form of a table, including: trustworthy bounding box classification, predicted bounding box classification, and directed edge vector values ​​(including scene information vectors and visual position information).

[0066] Based on the data in the table, the vector values ​​of directed edges with the same start and end are averaged and updated in the graph according to the following formula.

[0067] e = β·e old +(1-β)·e'

[0068] In the formula, β is the update rate adjustment factor, used to control the update rate of data in the map. It needs to be manually adjusted according to the actual scenario, and its value is generally in the range of 0.05-0.15. old e' is the vector value of the corresponding directed edge in the original graph, e' is the new vector value calculated according to the table, and e is the vector value of the corresponding directed edge in the updated graph.

[0069] Experimental setup: A dataset was created by capturing images from real-world scenes and manually labeling them. The following two tasks were then tested: 1) Object recognition task: using manually labeled bounding boxes to identify object categories; 2) using bounding boxes detected by the model to classify objects.

[0070] Table 1 Experimental Results

[0071]

[0072] As shown in Table 1, this embodiment can improve the accuracy of target detection and recognition in real-world scenarios, thereby enhancing the accuracy of the task.

[0073] Example 2

[0074] This embodiment provides a knowledge-enhanced multi-service robot object category recognition system, which includes:

[0075] The preliminary prediction module is used to acquire scene images of the robot's operation, extract visual information feature maps from the scene images, obtain preliminary prediction results of target detection boxes and their object categories in the scene images, and convert the visual information feature maps into scene information vectors.

[0076] The query prediction module is used to select trusted target label nodes in a pre-constructed knowledge graph based on the preliminary prediction results of the above object categories, form a matching vector based on the transformed scene information vector and target detection box information, calculate the similarity between the matching vector and the edge starting from the trusted target label node, and determine the query prediction result of the object category in the scene image based on the similarity.

[0077] The prediction fusion module is used to fuse the query prediction results of the object category in the scene image with the preliminary prediction results to obtain the final object category recognition result.

[0078] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0079] Example 3

[0080] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the knowledge-enhanced multi-service robot object category recognition method and system described above.

[0081] Example 4

[0082] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the knowledge-enhanced multi-service robot object category recognition method and system described above.

[0083] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A knowledge-enhanced method for object category recognition in multi-service robots, characterized in that, include: Acquire scene images of robot operation, extract visual information feature maps from the scene images, obtain preliminary prediction results of target detection boxes and their object categories in the scene images, and convert the visual information feature maps into scene information vectors; calculate the classification confidence of the preliminary prediction results of object categories in the scene images, save the object category target boxes whose classification confidence of the preliminary results is higher than a set threshold, and update the knowledge graph; During the process of updating the knowledge graph, the vector value of the corresponding directed edge in the updated knowledge graph is: the weighted sum of the first vector value and the second vector value; The first vector value is the vector value of the corresponding directed edge in the original knowledge graph, and its weight is a preset update speed adjustment factor; The second vector value is the scene information vector value corresponding to the preliminary prediction result of the object category in the scene image, and its weight is the difference between 1 and the preset update speed adjustment factor. The knowledge graph for identification consists of directed edges and nodes, where nodes represent the label values ​​of the identified objects, and the starting point of the directed edges is the object with high certainty, and the ending point is the inferred object. Based on the preliminary prediction results of the above object categories, trusted target label nodes in the pre-constructed knowledge graph are selected. A matching vector is formed based on the transformed scene information vector and the target detection box information. The similarity between the matching vector and the edge starting from the trusted target label node is calculated. The query prediction result of the object category in the scene image is determined according to the similarity. The query prediction results of object categories in the scene image are fused with the preliminary prediction results to obtain the final object category recognition result.

2. The knowledge-enhanced multi-service robot object category recognition method as described in claim 1, characterized in that, The final object category recognition result is the weighted sum of the query prediction result and the preliminary prediction result of the object category in the scene image.

3. The knowledge-enhanced multi-service robot object category recognition method as described in claim 1, characterized in that, The visual information feature map is converted into a scene information vector through a fully connected layer.

4. The knowledge-enhanced multi-service robot object category recognition method as described in claim 1, characterized in that, The similarity is characterized by cosine similarity.

5. A knowledge-enhanced multi-service robot object category recognition system, characterized in that, include: The preliminary prediction module is used to acquire scene images of robot operation, extract visual information feature maps from the scene images, obtain preliminary prediction results of target detection boxes and their object categories in the scene images, and convert the visual information feature maps into scene information vectors; calculate the classification confidence of the preliminary prediction results of object categories in the scene images, save the object category target boxes whose classification confidence of the preliminary results is higher than a set threshold, and update the knowledge graph. During the process of updating the knowledge graph, the vector value of the corresponding directed edge in the updated knowledge graph is: the weighted sum of the first vector value and the second vector value; The first vector value is the vector value of the corresponding directed edge in the original knowledge graph, and its weight is a preset update speed adjustment factor; The second vector value is the scene information vector value corresponding to the preliminary prediction result of the object category in the scene image, and its weight is the difference between 1 and the preset update speed adjustment factor. The knowledge graph for identification consists of directed edges and nodes, where nodes represent the label values ​​of the identified objects, and the starting point of the directed edges is the object with high certainty, and the ending point is the inferred object. The query prediction module is used to select trusted target label nodes in a pre-constructed knowledge graph based on the preliminary prediction results of the above object categories, form a matching vector based on the transformed scene information vector and target detection box information, calculate the similarity between the matching vector and the edge starting from the trusted target label node, and determine the query prediction result of the object category in the scene image based on the similarity. The prediction fusion module is used to fuse the query prediction results of the object category in the scene image with the preliminary prediction results to obtain the final object category recognition result.

6. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the knowledge-enhanced multi-service robot object category recognition method as described in any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the knowledge-enhanced multi-service robot object category recognition method as described in any one of claims 1-4.

Images