A three-dimensional point cloud scene target positioning method based on natural language instructions

Abstract

The application relates to the field of artificial intelligence and computer vision, and provides a three-dimensional point cloud scene target positioning method based on a natural language instruction, which comprises the following steps: screening a relation triple containing an object category in a three-dimensional visual scene; determining a hierarchical attribute of the object category and constructing a hierarchical knowledge graph; analyzing a given natural language description to obtain a plurality of two-order semantic triples; adding the hierarchical attribute to the object in the two-order semantic triple to obtain a hierarchical two-order semantic triple; determining an initial visual feature of an object in a three-dimensional point cloud scene to be recognized; outputting a prediction score of the object; screening the objects corresponding to the prediction scores from high to low to obtain candidate objects; outputting a matching score of the candidate objects and a first high-order semantic triple; and determining a target object in the three-dimensional point cloud scene to be recognized. The application improves the prediction accuracy and has stronger robustness.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision, specifically to a method for localizing a 3D point cloud scene target based on natural language instructions. Background Technology

[0002] In recent years, with the rapid development of intelligent robots and human-computer interaction in real-world scenarios, research on target localization within 3D point cloud scenes based on natural language descriptions has attracted increasing attention. Humans issue commands to mobile robots using natural language, and the robots locate target objects in their 3D scenes based on the natural language descriptions, significantly improving the intelligence level of mobile robots.

[0003] Currently, target localization in 3D point cloud scenes based on natural language descriptions faces challenges such as reducing the model's over-reliance on target object labels, abstracting free-form language description relationship features, and fusing natural language and 3D point cloud information across modalities. Current text-guided 3D point cloud target detection methods still heavily rely on target object labels; however, this label data is not only expensive but also requires significant manual annotation time. Furthermore, current methods fail to abstract the complex and irregular natural language into a clear structure and align the natural language features within that structure with the 3D point cloud scene at a fine-grained cross-modal level. These shortcomings limit the accuracy of 3D target localization and hinder mobile robots from understanding human natural language more intelligently and performing subsequent tasks accordingly. Summary of the Invention

[0004] The technical problems solved by this invention are how to reduce the model's over-reliance on target object labels, how to abstract free-style language description relationship features, and how to fuse natural language and 3D point cloud information across modalities.

[0005] To address the above problems, this invention provides a method for localizing a 3D point cloud scene based on natural language instructions, comprising:

[0006] Filter out relation triples that contain object categories in a 3D visual scene from the knowledge graph data source;

[0007] Determine the hierarchical attributes of the object category, and construct a hierarchical knowledge graph based on the relation triples and the hierarchical attributes of the object category;

[0008] Parse a given natural language description to obtain multiple two-order semantic triples;

[0009] Based on the hierarchical knowledge graph, hierarchical attributes are added to the objects in the two-order semantic triples to obtain hierarchical two-order semantic triples; the hierarchical two-order semantic triples include a first low-order semantic triple and a first high-order semantic triple.

[0010] Extract the first visual features, spatial features, semantic features, hierarchical attributes, and prior knowledge features of objects in the three-dimensional point cloud scene to be identified; and determine the initial visual features of the objects in the three-dimensional point cloud scene to be identified based on the first visual features, the spatial features, the semantic features, the hierarchical attributes, and the prior knowledge features.

[0011] The initial visual features and the first low-order semantic triple are input into the constructed multi-level attention model, and the predicted score of the object is output.

[0012] From the predicted scores of the objects, multiple objects corresponding to the predicted scores are selected from high to low as candidate objects.

[0013] The candidate object and the first higher-order semantic triple are input into a trained higher-order semantic triple alignment network, and the matching score between the candidate object and the first higher-order semantic triple is output; the target object in the 3D point cloud scene to be identified is determined based on the matching score.

[0014] Furthermore, the step of inputting the initial visual features and the first low-order semantic triples into the constructed multi-level attention model, and outputting the prediction score of the object, includes:

[0015] The initial visual features are input into the Self-Attention model, which outputs the second visual features.

[0016] Input the first low-order semantic triplet into the Self-Attention model to output language features;

[0017] The second visual feature and the language feature are sequentially input into the Cross-Attention model and the multilayer perceptron, and the predicted score of the object is output.

[0018] Furthermore, the training of the multi-level attention model includes:

[0019] Multiple visual negative samples and multiple language negative samples are generated based on the positive samples; the positive samples include the training 3D point cloud scene and the corresponding second low-order semantic triplet.

[0020] Construct a visual objective function based on the positive samples and the visual negative samples;

[0021] Construct a language objective function based on the positive samples and the negative language samples;

[0022] The parameters of the multi-level attention model are optimized based on the visual objective function and the language objective function.

[0023] Furthermore, the step of generating multiple visual negative samples and multiple language negative samples based on positive samples includes:

[0024] Obtain the second higher-order semantic triplet corresponding to the training 3D point cloud scene;

[0025] The objects in the training 3D point cloud scene that have the same hierarchical attributes as the target object units in the second higher-order semantic triplet are deleted to obtain visual negative samples.

[0026] Using the hierarchical knowledge graph, a substitute object is selected, and the target object unit in the second higher-order semantic triple is replaced with the substitute item to obtain a language negative sample; the substitute item does not exist in the training 3D point cloud scene; the substitute item has the same hierarchical attribute as the target object unit in the second higher-order semantic triple.

[0027] Further, the step of inputting the candidate object and the first higher-order semantic triplet into the trained higher-order semantic triplet alignment network, and outputting the matching score between the candidate object and the first higher-order semantic triplet includes:

[0028] Based on whether the candidate object and the target object unit in the first higher-order semantic triple are of the same level attribute, the fully connected network determines the first matching score between the candidate object and the target object unit in the first higher-order semantic triple.

[0029] Based on whether the reference object of the candidate object is at the same level as the reference object unit in the first higher-order semantic triple, the fully connected network determines the second matching score between the reference object of the candidate object and the reference object unit in the first higher-order semantic triple.

[0030] The third matching score is determined based on the candidate object, the target object unit and the reference object unit in the first higher-order semantic triplet, and the fully connected network.

[0031] The matching score between the candidate object and the first higher-order semantic triple is obtained based on the first matching score, the second matching score, and the third matching score.

[0032] Furthermore, the extraction of the first visual features, spatial features, semantic features, hierarchical attributes, and prior knowledge features of objects in the 3D point cloud scene to be identified includes:

[0033] The 3D point cloud scene to be identified is input into the PointNet++ network, and multiple initial candidate boxes are generated using the VoteNet voting strategy.

[0034] Based on the initialized candidate boxes, extract the first visual features, spatial features, semantic features, and hierarchical attributes of the corresponding objects;

[0035] The hierarchical knowledge graph is input into a graph convolutional network to determine the prior knowledge features of the objects in the 3D point cloud scene to be identified.

[0036] Furthermore, the step of extracting the first visual features, spatial features, semantic features, and hierarchical attributes of the corresponding object based on the initialized candidate box includes:

[0037] The first visual features of the object corresponding to the initial candidate box are extracted using the average pooling method.

[0038] The absolute spatial position of the initial candidate box is encoded to obtain the spatial features of the corresponding object;

[0039] The word embedding vectors of the object categories corresponding to the initialization candidate boxes are obtained using GloVe, and used as the semantic features of the corresponding objects;

[0040] The hierarchical attributes of the objects corresponding to the initialization candidate boxes are obtained based on the hierarchical attributes of the object categories.

[0041] Furthermore, the hierarchical attribute for determining the object category includes:

[0042] Input every two objects in the 3D visual scene into a trained binary classifier, and output whether there is a support relationship between the objects.

[0043] The hierarchical attributes of object categories are determined based on whether there is a support relationship between objects.

[0044] Furthermore, the binary classifier includes an activation function and a multilayer perceptron.

[0045] Furthermore, the construction of a hierarchical knowledge graph based on the relation triples and the hierarchical attributes of the object categories includes:

[0046] Determine the word embedding vector of the object category based on the object category;

[0047] The embedding vector of the hierarchical attribute is determined based on the hierarchical attribute of the object category;

[0048] A hierarchical knowledge graph is constructed based on the relation triples, the word embedding vectors of the object categories, the hierarchical attributes of the object categories, and the embedding vectors of the hierarchical attributes.

[0049] The beneficial effects of the 3D point cloud scene target localization method based on natural language commands described in this invention are:

[0050] First, this invention performs targeted analysis on natural language descriptions, using analysis to abstract complex and irregular natural language into a fixed structure in the form of two-stage semantic triples, thereby realizing the structuring of natural language.

[0051] Secondly, this invention abstracts the spatial hierarchy attributes of objects in 3D point cloud scenes from a large dataset of natural language describing 3D scenes, and integrates these spatial hierarchy attributes with the 3D point cloud scene, natural language, and knowledge graph. Introducing spatial hierarchy attributes can, on the one hand, compensate for the accuracy loss caused by the lack of target object labels in weak supervision, and overcome the over-reliance on target object labels in existing works; on the other hand, it utilizes hierarchical prior knowledge to enhance the visual feature representation of objects in the scene, improving algorithm performance and efficiency while exhibiting stronger robustness. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a method for localizing a 3D point cloud scene target based on natural language instructions, according to an embodiment of the present invention. Detailed Implementation

[0053] The present invention will now be described in detail with reference to the accompanying drawings.

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] This invention provides a method for target localization in a 3D point cloud scene based on natural language commands, such as... Figure 1 As shown, the method includes the following steps:

[0056] 101. Filter out relation triples that contain object categories in a 3D visual scene from the knowledge graph data source.

[0057] Specifically, the publicly available knowledge graph data source ConceptNet was selected, and data filtering and cleaning were performed on ConceptNet. First, common object categories in 3D visual scenes were extracted from the SUNRGBD, Scannet, and 3RScan datasets, and then relation triples containing only these common object categories were filtered from ConceptNet. The filtered ConceptNet is defined as K. e ={V e E e}, node set V eE contains common object categories in 3D visual scenes, and a set of directed edges. e This represents the relationships between common object categories, including positional relationships and subordinate relationships, indicating coarse-grained relationships.

[0058] 102. Determine the hierarchical attributes of the object category, and construct a hierarchical knowledge graph based on the relation triples and the hierarchical attributes of the object category.

[0059] In one possible implementation, the hierarchical attribute for determining the object category includes:

[0060] Input every two objects in the 3D visual scene into a trained binary classifier, and output whether there is a support relationship between the objects.

[0061] The hierarchical attributes of object categories are determined based on whether there is a support relationship between objects.

[0062] In one possible implementation, the binary classifier includes an activation function and a multilayer perceptron.

[0063] Specifically, let's first illustrate the hierarchical attributes of object categories. For a set of object categories in a dataset, a multilayer perceptron is used to determine whether there is a support relationship between objects. Let's assume that the first layer of objects is only supported by the ground, the second layer of objects is supported by the first layer of objects, the second layer of objects provides support for the third layer of objects, and so on, to determine the hierarchical attributes of each object in the set of object categories.

[0064] Determining the hierarchical attributes of object categories specifically involves: firstly, collecting language datasets describing the spatial relationships between objects in a 3D scene from 3DSSG, Nr3D, and Sr3D; and then using an offline language parser to parse the natural language into semantic triples, {obj α ,R αβ ,obj β}, where obj α and obj β Representing two objects, R αβ The spatial relation words are used to determine whether a support relationship exists between two objects. These words are then used to binarize and label the objects. Next, a binary classifier is trained using this labeled data to determine whether support relationships exist between objects in the object category set.

[0065] p αβ =sigmoid(MLP([o α ·o β ])

[0066] In the formula, p αβThe prediction score represents the presence or absence of a support relationship between objects; sigmoid represents the activation function; MLP represents a multilayer perceptron; o α o β These represent objects whose support relationship is predicted to exist. GloVe encodes these two objects into feature vectors, with · indicating the vector concatenation method. If the prediction score for a support relationship between the two objects is greater than a preset threshold, a support relationship is determined to exist between the two objects; otherwise, no support relationship is determined to exist. For example, the threshold can be set to 0.5.

[0067] The trained binary classifier can be obtained by following the above method. The object categories in the 3D visual scene are input into the trained binary classifier. The trained binary classifier can determine whether there is a support relationship between the object categories in the 3D visual scene, thereby determining the hierarchical attributes of the object categories.

[0068] In one possible implementation, constructing a hierarchical knowledge graph based on the relation triples and the hierarchical attributes of the object category includes:

[0069] Determine the word embedding vector of the object category based on the object category;

[0070] The embedding vector of the hierarchical attribute is determined based on the hierarchical attribute of the object category;

[0071] A hierarchical knowledge graph is constructed based on the relation triples, the word embedding vectors of the object categories, the hierarchical attributes of the object categories, and the embedding vectors of the hierarchical attributes.

[0072] Specifically, the GloVe word embedding vector of the object category is determined based on the object category, and the embedding vector of the hierarchical attribute is determined based on the hierarchical attribute of the object category. Based on the relation triple, the GloVe word embedding vector of the object category and the embedding vector of the hierarchical attribute are concatenated according to the hierarchical attribute of the object category to represent the embedding vector of each object node in the knowledge graph in the high-dimensional space.

[0073] 103. Parse the given natural language description to obtain multiple two-order semantic triples.

[0074] Specifically, the two-order semantic triples here refer to the initial two-order semantic triples. Given an unordered natural language description, an offline language parser parses multiple two-order semantic triples. The low-order semantic triples contain three semantic units: a target object unit, a spatial relation word unit, and a reference object unit. The high-order semantic triples also contain three semantic units: a target object unit, a spatial relation word unit, and a reference object unit. The reference object unit is a unit related to finding a reference object for the target object; for example, if you want to find a water glass on a table, the target object is the water glass, and the reference object is the table. The spatial relation word unit in the high-order semantic triples is the same as that in the low-order semantic triples. Compared to the target object unit and reference object unit in the low-order semantic triples, the target object unit and reference object unit add adjectives for the object noun, including color, material, and size. Then, GloVe is used to encode each semantic unit to obtain the initial two-order semantic triples. By performing targeted parsing of the natural language description, the spatial positional relationship features between the target object and the reference object in the natural language description are efficiently extracted.

[0075] 104. Add hierarchical attributes to the objects in the two-order semantic triples according to the hierarchical knowledge graph to obtain hierarchical two-order semantic triples; the hierarchical two-order semantic triples include a first low-order semantic triple and a first high-order semantic triple.

[0076] Based on the hierarchical knowledge graph, hierarchical attributes are added to the objects in the two-order semantic triples. That is, hierarchical information is added to the target object unit and the reference object unit in the two-order semantic triples to obtain hierarchical two-order semantic triples.

[0077] 105. Extract the first visual features, spatial features, semantic features, hierarchical attributes, and prior knowledge features of the objects in the three-dimensional point cloud scene to be identified, and determine the initial visual features of the objects in the three-dimensional point cloud scene to be identified based on the first visual features, the spatial features, the semantic features, the hierarchical attributes, and the prior knowledge features.

[0078] In one possible implementation, the extraction of the first visual features, spatial features, semantic features, hierarchical attributes, and prior knowledge features of objects in the 3D point cloud scene to be identified includes:

[0079] The 3D point cloud scene to be identified is input into the PointNet++ network, and multiple initial candidate boxes are generated using the VoteNet voting strategy.

[0080] Based on the initialized candidate boxes, extract the first visual features, spatial features, semantic features, and hierarchical attributes of the corresponding objects;

[0081] The hierarchical knowledge graph is input into a graph convolutional network to determine the prior knowledge features of the objects in the 3D point cloud scene to be identified.

[0082] In one possible implementation, the step of extracting the first visual features, spatial features, semantic features, and hierarchical attributes of the corresponding object based on the initialized candidate box includes:

[0083] The first visual features of the object corresponding to the initial candidate box are extracted using the average pooling method.

[0084] The absolute spatial position of the initial candidate box is encoded to obtain the spatial features of the corresponding object;

[0085] The word embedding vectors of the object categories corresponding to the initialization candidate boxes are obtained using GloVe, and used as the semantic features of the corresponding objects;

[0086] The hierarchical attributes of the objects corresponding to the initialization candidate boxes are obtained based on the hierarchical attributes of the object categories.

[0087] Specifically, PointNet++ is used as the backbone point cloud feature extraction network. The 3D point cloud scene to be identified is input into PointNet++, and 256 initial candidate boxes are generated using the VoteNet voting strategy. Each initial candidate box corresponds to an object, i.e., an object instance. The initial candidate box is represented as {v, d}, where v = (d... x ,d y ,d z ,l x ,l y ,l z ), d x ,d y ,d z Let l be the center coordinates of the bounding box. x ,l y ,l zLet represent the lengths of the three coordinate axes of the bounding box, and ...

[0088] 106. Input the initial visual features and the first low-order semantic triple into the constructed multi-level attention model, and output the prediction score of the object.

[0089] In one possible implementation, the step of inputting the initial visual features and the first low-order semantic triples into the constructed multi-level attention model and outputting the prediction score of the object includes:

[0090] The initial visual features are input into the Self-Attention model, which outputs the second visual features.

[0091] Input the first low-order semantic triplet into the Self-Attention model to output language features;

[0092] The second visual feature and the language feature are sequentially input into the Cross-Attention model and the multilayer perceptron, and the predicted score of the object is output.

[0093] Specifically, multi-level attention model as follows:

[0094]

[0095] In the formula, (O,T) l () represents a positive sample, MLP represents a multilayer perceptron, and CrossAttn represents a cross-attention model. In Indicates second visual features, Indicates language features, This represents the predicted score for an object.

[0096] The multi-level attention model includes: inputting initial visual features and a first low-order semantic triple into a Self-Attention model to extract features within the visual and linguistic unimodals, respectively, to obtain visual and linguistic features, with the visual features serving as the second visual feature. The second visual feature and linguistic features are then input into a Cross-Attention model and a multilayer perceptron, outputting a prediction score for each object.

[0097] In this embodiment of the invention, the accuracy of locating target objects in a scene is improved by aligning multimodal features, namely visual features and linguistic features.

[0098] In one possible implementation, training the multi-level attention model includes:

[0099] Multiple visual negative samples and multiple language negative samples are generated based on the positive samples; the positive samples include the training 3D point cloud scene and the corresponding second low-order semantic triplet.

[0100] Construct a visual objective function based on the positive samples and the visual negative samples;

[0101] Construct a language objective function based on the positive samples and the negative language samples;

[0102] The parameters of the multi-level attention model are optimized based on the visual objective function and the language objective function.

[0103] Specifically, due to the unique characteristics of 3D target localization, directly sampling negative samples in the smallest batch using traditional methods would result in "false negative samples." Therefore, this invention constructs negative samples with hierarchical discrimination to ensure cross-modal consistency of negative samples. The negative samples constructed in this invention include visual negative samples and language negative samples.

[0104] In this embodiment of the invention, a minimization objective function L is defined. infonce (θ) is used to maximize the lower bound of mutual information between objects in the scene and the first low-order semantic triples:

[0105]

[0106] In the formula, MI(O,T) l ) represents object O and the first low-order semantic triple T. l Mutual information between them, K represents the number of positive samples in a batch, for example, the number of positive samples can be set to 32, and the objective function L infonce (θ) is derived from the visual objective function L visual (θ) and the language objective function L Iang Composed of (θ), where θ represents the visual target function Lvisual (θ) and the language objective function L lang (θ) Parameters of the supervised multi-level attention model.

[0107] Positive samples include the training 3D point cloud scene and the corresponding second low-order semantic triples. For a given positive sample (O, T) in a batch... l Generate K visual negative samples, with visual objective function L. visual (θ) is:

[0108]

[0109] In the formula, This represents a multi-level attention model. Let represent the i-th visual negative sample, and n represent a negative number, with i ranging from 1 to K.

[0110] For a given positive sample (O, T) in Batch l Generate K negative language samples, with the language objective function L. lang (θ) is:

[0111]

[0112] In the formula, Let i represent the i-th negative language sample, where i ranges from 1 to K.

[0113] The parameters of the multi-level attention model are optimized based on the constructed visual and linguistic objective functions.

[0114] In one possible implementation, generating multiple visual negative samples and multiple language negative samples based on positive samples includes:

[0115] Obtain the second higher-order semantic triplet corresponding to the training 3D point cloud scene;

[0116] The objects in the training 3D point cloud scene that have the same hierarchical attributes as the target object units in the second higher-order semantic triplet are deleted to obtain visual negative samples.

[0117] Using the hierarchical knowledge graph, a substitute object is selected, and the target object unit in the second higher-order semantic triple is replaced with the substitute item to obtain a language negative sample; the substitute item does not exist in the training 3D point cloud scene; the substitute item has the same hierarchical attribute as the target object unit in the second higher-order semantic triple.

[0118] Specifically, for visual negative samples, maintain the same level as positive samples (O, T) lFor positive samples, multiple alternative objects at different levels are selected from the hierarchical knowledge graph to replace the objects in the target object unit of the second higher-order semantic triple in the positive sample. These alternative objects must satisfy the following two conditions:

[0119] First, the object does not exist in the scene;

[0120] Second, the objects in the replacement object and the target object unit must be at the same level and have the same supporting object.

[0121] By satisfying the above two conditions, we can ensure that the newly constructed semantic triples are logically consistent.

[0122] The above processing can achieve alignment of low-level semantics.

[0123] In this embodiment of the invention, by constructing negative samples with hierarchical discrimination, the problem of false negative samples in the direct application of contrastive learning to target localization in 3D point cloud scenes is solved. This effectively overcomes the bottleneck of cross-modal feature domain differences between natural language and 3D point clouds, and overcomes the cross-modal semantic inconsistency problem caused by randomly selecting negative samples from sample data. This allows the embodiments of the invention to accurately locate target objects in complex 3D point cloud scenes based on an understanding of natural language.

[0124] 107. Select multiple objects with prediction scores from high to low from the predicted scores of the objects, and use them as candidate objects.

[0125] Specifically, based on the alignment of low-order semantic triples, it is also necessary to exclude interfering objects among similar objects according to the hierarchical attributes in the first high-order semantic triple. Interfering objects among similar objects have the same hierarchical attributes as the target object, so only objects at the same level as the target object unit in the first low-order semantic triple need to be considered, rather than all layers.

[0126] First, select multiple objects with the highest and lowest prediction scores from the object prediction scores as candidate objects, as follows:

[0127]

[0128] In the formula, o F Let F represent the candidate object, and let argtopk represent the k highest-scoring objects selected as candidates using the argtopk sorting function.

[0129] 108. Input the candidate object and the first higher-order semantic triple into the trained higher-order semantic triple alignment network, and output the matching score between the candidate object and the first higher-order semantic triple; determine the target object in the three-dimensional point cloud scene to be identified based on the matching score.

[0130] In one possible implementation, the step of inputting the candidate object and the first higher-order semantic triplet into a trained higher-order semantic triplet alignment network, and outputting a matching score between the candidate object and the first higher-order semantic triplet, includes:

[0131] Based on whether the candidate object and the target object unit in the first higher-order semantic triple are of the same level attribute, the fully connected network determines the first matching score between the candidate object and the target object unit in the first higher-order semantic triple.

[0132] Based on whether the reference object of the candidate object is at the same level as the reference object unit in the first higher-order semantic triple, the fully connected network determines the second matching score between the reference object of the candidate object and the reference object unit in the first higher-order semantic triple.

[0133] The third matching score is determined based on the candidate object, the target object unit and the reference object unit in the first higher-order semantic triplet, and the fully connected network.

[0134] The matching score between the candidate object and the first higher-order semantic triple is obtained based on the first matching score, the second matching score, and the third matching score.

[0135] Specifically, further matching is performed between candidate objects in the scene and target object units and reference object units in the first higher-order semantic triples:

[0136]

[0137] In the formula, This represents the first matching score between the candidate object and the target object unit in the first higher-order semantic triplet. F represents the second matching score between the reference object of the candidate object and the reference object unit in the first higher-order semantic triple. t and F a Both represent fully connected networks, and ⊕ indicates a characteristic linking method. This represents the target object unit in the first higher-order semantic triplet. This represents the reference object unit in the first higher-order semantic triple. This indicates whether the candidate object and the target object are at the same level; in other words, whether the candidate object is at the same level as the target object. This indicates whether the candidate object and the reference object are at the same level. This indicates the hierarchical attribute marker corresponding to the target object unit. This indicates the hierarchical attribute marker corresponding to the reference object unit.

[0138] Next, the degree of matching between the relationship between candidate objects and reference objects and spatial relationship units is obtained, and this degree of matching is represented by the second matching score:

[0139]

[0140] In the formula, F represents the second match score. r Indicates a fully connected network. Both ⊕ and ⊕ indicate feature linking methods. Represents candidate object o i and its reference object o j They are at the same level as the target object unit and the reference object unit in the first higher-order semantic triple, respectively. Being at the same level means that they have the same hierarchical attributes. This refers to the spatial relational word unit in the first higher-order semantic triplet.

[0141] The matching scores between the candidate object and the first higher-order semantic triple are obtained based on the first matching score, the second matching score, and the third matching score, as follows:

[0142]

[0143] In the formula, ρ represents the matching score between the candidate object and the first higher-order semantic triple, where ρ represents the number of triples.

[0144] Among all the matching scores between candidate objects and the first higher-order semantic triple, the candidate object with the highest matching score is the predicted target object.

[0145] The high-order semantic triple alignment network model is supervised by reconstructing visual features and minimizing the distance between visual and semantic features:

[0146]

[0147]

[0148]

[0149] In the formula, c∈{t,a,r}, R c Indicates a fully connected network. The specific meaning is determined by the letter 'c'. If c = t, then... This indicates that the model predicts the combination of visual features of the target object. If c = a, then... This indicates that the model predicts the combination of visual features of the reference object. If c = r, then... This indicates that the model predicts a combination of spatial relationship features between the target object and the reference object. Similarly, if c = t, This represents the visual features of the reconstructed target object. If c = a, Represents the visual features reconstructed from the reference object; if c = r, L represents the visual features after spatial relationship reconstruction. rec This represents the sum of losses for all units, where o is the sum of losses for all units. i Represents the visual characteristics of the target object, o j It represents the visual characteristics of the reference object.

[0150] This article uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for localizing a target in a 3D point cloud scene based on natural language commands, characterized in that, include: Filter out relation triples that contain object categories in a 3D visual scene from the knowledge graph data source; Determine the hierarchical attributes of the object category, and construct a hierarchical knowledge graph based on the relation triples and the hierarchical attributes of the object category; Parse a given natural language description to obtain multiple two-order semantic triples; Based on the hierarchical knowledge graph, hierarchical attributes are added to the objects in the two-order semantic triples to obtain hierarchical two-order semantic triples; the hierarchical two-order semantic triples include a first low-order semantic triple and a first high-order semantic triple. Extract the first visual features, spatial features, semantic features, hierarchical attributes, and prior knowledge features of objects in the three-dimensional point cloud scene to be identified; and determine the initial visual features of the objects in the three-dimensional point cloud scene to be identified based on the first visual features, the spatial features, the semantic features, the hierarchical attributes, and the prior knowledge features. The initial visual features and the first low-order semantic triple are input into the constructed multi-level attention model, and the predicted score of the object is output. From the predicted scores of the objects, multiple objects corresponding to the predicted scores are selected from high to low as candidate objects. The candidate object and the first higher-order semantic triple are input into a trained higher-order semantic triple alignment network, and the matching score between the candidate object and the first higher-order semantic triple is output; the target object in the 3D point cloud scene to be identified is determined based on the matching score.

2. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 1, characterized in that, The process of inputting the initial visual features and the first low-order semantic triples into the constructed multi-level attention model, and outputting the prediction score of the object, includes: The initial visual features are input into the Self-Attention model, which outputs the second visual features. Input the first low-order semantic triplet into the Self-Attention model to output language features; The second visual feature and the language feature are sequentially input into the Cross-Attention model and the multilayer perceptron, and the predicted score of the object is output.

3. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 2, characterized in that, The training of the multi-level attention model includes: Multiple visual negative samples and multiple language negative samples are generated based on the positive samples; the positive samples include the training 3D point cloud scene and the corresponding second low-order semantic triplet. Construct a visual objective function based on the positive samples and the visual negative samples; Construct a language objective function based on the positive samples and the negative language samples; The parameters of the multi-level attention model are optimized based on the visual objective function and the language objective function.

4. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 3, characterized in that, The step of generating multiple visual negative samples and multiple language negative samples based on positive samples includes: Obtain the second higher-order semantic triplet corresponding to the training 3D point cloud scene; The objects in the training 3D point cloud scene that have the same hierarchical attributes as the target object units in the second higher-order semantic triplet are deleted to obtain visual negative samples. Using the hierarchical knowledge graph, a substitute object is selected, and the target object unit in the second higher-order semantic triple is replaced with the substitute item to obtain a language negative sample; the substitute item does not exist in the training 3D point cloud scene; the substitute item has the same hierarchical attribute as the target object unit in the second higher-order semantic triple.

5. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 1, characterized in that, The step of inputting the candidate object and the first higher-order semantic triplet into a trained higher-order semantic triplet alignment network, and outputting the matching score between the candidate object and the first higher-order semantic triplet includes: Based on whether the candidate object and the target object unit in the first higher-order semantic triple are of the same level attribute, the fully connected network determines the first matching score between the candidate object and the target object unit in the first higher-order semantic triple. Based on whether the reference object of the candidate object is at the same level as the reference object unit in the first higher-order semantic triple, the fully connected network determines the second matching score between the reference object of the candidate object and the reference object unit in the first higher-order semantic triple. The third matching score is determined based on the candidate object, the target object unit and the reference object unit in the first higher-order semantic triplet, and the fully connected network. The matching score between the candidate object and the first higher-order semantic triple is obtained based on the first matching score, the second matching score, and the third matching score.

6. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 1, characterized in that, The extraction of the first visual features, spatial features, semantic features, hierarchical attributes, and prior knowledge features of objects in the 3D point cloud scene to be identified includes: The 3D point cloud scene to be identified is input into the PointNet++ network, and multiple initial candidate boxes are generated using the VoteNet voting strategy. Based on the initialized candidate boxes, extract the first visual features, spatial features, semantic features, and hierarchical attributes of the corresponding objects; The hierarchical knowledge graph is input into a graph convolutional network to determine the prior knowledge features of the objects in the 3D point cloud scene to be identified.

7. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 6, characterized in that, The step of extracting the first visual features, spatial features, semantic features, and hierarchical attributes of the corresponding object based on the initialized candidate box includes: The first visual features of the object corresponding to the initial candidate box are extracted using the average pooling method. The absolute spatial position of the initial candidate box is encoded to obtain the spatial features of the corresponding object; The word embedding vectors of the object categories corresponding to the initialization candidate boxes are obtained using GloVe, and used as the semantic features of the corresponding objects; The hierarchical attributes of the objects corresponding to the initialization candidate boxes are obtained based on the hierarchical attributes of the object categories.

8. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 1, characterized in that, The hierarchical attributes for determining the object category include: Input every two objects in the 3D visual scene into a trained binary classifier, and output whether there is a support relationship between the objects. The hierarchical attributes of object categories are determined based on whether there is a support relationship between objects.

9. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 8, characterized in that, The binary classifier includes an activation function and a multilayer perceptron.

10. The method for localizing a 3D point cloud scene based on natural language commands as described in claim 1, characterized in that, The construction of a hierarchical knowledge graph based on the relation triples and the hierarchical attributes of the object category includes: Determine the word embedding vector of the object category based on the object category; The embedding vector of the hierarchical attribute is determined based on the hierarchical attribute of the object category; A hierarchical knowledge graph is constructed based on the relation triples, the word embedding vectors of the object categories, the hierarchical attributes of the object categories, and the embedding vectors of the hierarchical attributes.

Images