Visual sparse hybrid expert-based fingerprint representation method

Abstract

The invention discloses a body finger representation reaching method based on a visual sparse hybrid expert, and the method comprises the following steps: 1, constructing an end-to-end basic model which adopts a cross-modal coding structure based on a transformer layer, and comprises a language instruction encoder, a panoramic observation encoder and a cross-modal encoder; step 2, replacing FFN layers in the panoramic observation encoder and the cross-modal encoder with MoE layers; and step 3, constructing a routing network, and dynamically allocating expert networks to process different visual features according to the visual lexical elements and the category information. According to the method, collaborative optimization of navigation view angle selection and target object recognition tasks is realized through the multi-task processing capability and the dynamic routing mechanism of the hybrid expert model, and the problem of difficulty in view and object heterogeneous relationship modeling in language-guided navigation is effectively solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of embodied intelligence in the field of computer, and particularly relates to an embodied pointing representation method based on visual sparse mixed experts. BACKGROUND

[0002] It has been an important goal in the field of artificial intelligence and robotics to enable robots to complete tasks according to natural language instructions. In this direction, visual and language navigation has attracted more and more researchers' attention in recent years due to its wide range of real-world application scenarios. In the embodied pointing representation positioning task, the agent needs to select a discrete view according to the language instruction to determine the next navigation direction, and identify the target object among multiple candidate objects in the final position reached. However, the navigation view and the target object are not only different in distribution, but also serve different tasks, so it is crucial to build a unified framework that can model both the navigation view and the target object to solve the two core challenges of language-guided navigation and pointing representation understanding.

[0003] According to the good ability of the mixed expert model (MoE) to handle multiple input and multiple task problems, a unified framework can be designed to allocate the navigation view selection and target object identification tasks to different expert sub-networks for processing through a dynamic routing mechanism. Each expert module focuses on a specific task (i.e., view prediction or object localization), while the gating network adaptively fuses the outputs of each expert according to the input instruction and scene context. This architecture can effectively model the heterogeneous relationship between the navigation view and the target object, while improving computational efficiency through a sparse activation mechanism. SUMMARY

[0004] The purpose of the present application is to provide an embodied pointing representation method based on visual sparse mixed experts, and to design a unified framework for language-guided navigation and pointing representation understanding based on the mixed expert model (MoE), which optimizes the navigation view selection and target object identification tasks through dynamic task allocation and adaptive feature fusion, to solve the problems of existing methods, such as the difficulty in effectively modeling the heterogeneous relationship between the navigation view and the target object, low computational efficiency, and insufficient multi-task coordination optimization.

[0005] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0006] An embodied pointing representation method based on visual sparse mixed experts, comprising the following steps:

[0007] Step 1: Constructing an end-to-end base model, the base model adopts a cross-modal encoding structure based on transformer layers, including a language instruction encoder, a panoramic observation encoder, and a cross-modal encoder;

[0008] Step 2, replace FFN layers in panoramic observation encoder and cross-modal encoder with MoE layers;

[0009] Step 3, build routing network to dynamically assign expert networks to process different visual features according to visual tokens and category information.

[0010] Preferably, in step 1, the language instruction encoder is BERT.

[0011] Preferably, in step 1, the panoramic encoder is a 2-layer transformer.

[0012] Preferably, in step 1, the cross-modal encoder is a 4-layer transformer.

[0013] Preferably, in step 2, each MoE layer contains 4 expert networks.

[0014] Preferably, each expert network has the same settings as the original FFN layer.

[0015] Preferably, among the 4 expert networks of each MoE layer, at least one expert network is dedicated to navigation perspective feature extraction, and at least another expert network is dedicated to target object feature extraction.

[0016] Preferably, in step 3, the input of the routing network is I, which is the sum of visual tokens T and category information L represented by tokens, and I is processed by the routing network to obtain the assignment probability of each expert network, the two expert networks with the highest assignment probability are selected to process the visual tokens T, the corresponding probabilities P are obtained by applying the Softmax function to the two highest assignment probabilities, and the processing results of the two expert networks are weighted and summed according to the probabilities P to obtain new visual tokens T ' .

[0017] Preferably, the visual tokens T include navigation view tokens T n and target object tokens T o , and the corresponding category information L is navigation view category L n and target object category L o .

[0018] Preferably, the dynamic routing mechanism of the MoE layer achieves the following functions:

[0019] Assign navigation perspective prediction tasks to the first group of expert networks;

[0020] Assign target object recognition tasks to the second group of expert networks;

[0021] Only call 2 expert networks through sparse activation to reduce computational overhead.

[0022] Beneficial effects: compared with the prior art, the present application has the following advantages:

[0023] The present application proposes a language-guided navigation and pointing representation understanding unified framework based on a mixed expert model (MoE), which fully utilizes the advantages of multi-task learning and dynamic routing mechanism. The embodiment of the present application is a somatic pointing representation method based on visual sparse mixed experts, which uniformly processes the language-guided navigation and pointing representation understanding tasks. Through the heterogeneous task allocation mechanism of the MoE layer, the navigation view selection accuracy and target object recognition accuracy are optimized collaboratively. Through the sparse activation expert network allocation strategy, the model can efficiently model the heterogeneous relationship between the navigation view and the target object while maintaining a low computational overhead. Compared with the traditional multi-task learning method, the framework significantly improves the view prediction accuracy, target object recognition accuracy and inference speed, effectively solving the collaborative optimization problem of language-guided navigation and pointing representation understanding tasks. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flowchart of the somatic pointing representation method based on visual sparse mixed experts in the present application. DETAILED DESCRIPTION

[0025] The present application will be further described below in conjunction with the accompanying drawings.

[0026] Unless otherwise specified, all technical and scientific terms used in the present application have the same meaning as commonly understood by those skilled in the art to which the present application belongs. If the definition of a specific term disclosed in the prior art is inconsistent with any definition provided herein, the definition provided herein shall prevail.

[0027] transformer, a sequence feature encoding structure based on a self-attention mechanism, containing an encoder and a decoder stack structure, for extracting feature associations of input data;

[0028] FFN layer, full name Feed-Forward Network (Feed-Forward Network layer), is a feed-forward network layer in Transformer for processing single-point features independently;

[0029] MoE layer, full name Mixture of Experts (Mixture of Experts layer), is a mixed expert structure composed of multiple expert networks and a dynamic routing mechanism, only activating part of the experts to process the input;

[0030] BERT, full name Bidirectional Encoder Representations from Transformers, is a pre-training language understanding model based on Transformer encoder;

[0031] Visual word unit, the basic visual feature unit formed after the image is segmented;

[0032] The routing network dynamically assigns tasks to the gating module of the expert network based on input features;

[0033] Sparse activation, a mechanism that calls only a few expert networks to reduce computational overhead.

[0034] like Figure 1 As shown in FIG, the embodiment-referring expression method based on visual sparse hybrid experts includes the following steps:

[0035] Step 1: Build an end-to-end basic model;

[0036] In some preferred embodiments of the present invention, the basic model adopts a cross-modal encoding structure based on a transformer layer, including a language instruction encoder, a panoramic observation encoder, and a cross-modal encoder;

[0037] In some preferred embodiments of the present invention, the language instruction encoder is BERT, the panoramic encoder is a 2-layer transformer, and the cross-modal encoder is a 4-layer transformer.

[0038] Step 2: Replace the FFN layer in the panoramic observation encoder and the cross-modal encoder with the MoE layer;

[0039] In some preferred embodiments of the present invention, each MoE layer includes four expert networks, and each expert network has the same configuration as the original FFN layer.

[0040] In some preferred embodiments of the present invention, among the four expert networks in each MoE layer, at least one expert network is dedicated to navigation view feature extraction, and at least another expert network is dedicated to target object feature extraction.

[0041] In some preferred embodiments of the present invention, the dynamic routing mechanism of the MoE layer implements the following functions:

[0042] Assign the navigation viewpoint prediction task to the first group of expert networks;

[0043] Assign the target object recognition task to the second group of expert networks;

[0044] Through sparse activation, only two expert networks are called to reduce computational overhead.

[0045] Step 3: Build a routing network to dynamically assign expert networks to process different visual features based on visual word units and category information.

[0046] In some preferred embodiments of the present invention, the specific implementation steps of the routing network include: the input of the routing network is I, which is the sum of the visual word unit T and the category information L represented by the word unit, I passes through the routing network to obtain the corresponding distribution pair ratio of each expert network, selects the two expert networks with the highest distribution pair ratio to process the visual word unit T, and obtains the corresponding probability P for the two highest distribution pair ratios through the Softmax function. According to the probability P, the processing results of the two expert networks are weighted and summed to obtain a new visual word unit T. ' .

[0047] Among them, the visual word unit T includes the navigation view word unit T n and target object word T o , the corresponding category information L are navigation view category L n and target object category L o The navigation view word unit and the target object word unit correspond to the discrete viewing angle features of the environment that can be navigated and the visual features of the object to be identified, respectively.

[0048] The technical solutions of the present disclosure are further illustrated below with reference to specific examples. However, the following examples are merely examples of the present disclosure and do not represent or limit the scope of protection of the present disclosure. In the following examples, unless otherwise specified, all reagents and consumables used were purchased from common suppliers in the field, and the experimental methods and technical means used were conventional methods and means in the field.

[0049] Example

[0050] The embodiment-based embodied referential expression method of this embodiment includes the following steps:

[0051] (1) Build an end-to-end basic model;

[0052] Among them, the basic model adopts a cross-modal encoding structure based on the transformer layer, including:

[0053] Language instruction encoder, which uses BERT structure to process natural language instructions;

[0054] The panoramic observation encoder uses a two-layer Transformer structure to process panoramic visual observation;

[0055] The cross-modal encoder uses a 4-layer Transformer structure to fuse language and visual features.

[0056] (2) Replace the FFN layer in the panoramic observation encoder and cross-modal encoder with the MoE layer;

[0057] Each MoE layer contains 4 expert networks, each of which is set the same as the original FFN layer, and all expert networks retain the input and output dimensions and nonlinear activation functions of the original FFN layer.

[0058] Among the 4 expert networks of each MoE layer, at least one expert network is dedicated to navigation view feature extraction, and at least another expert network is dedicated to target object feature extraction.

[0059] The dynamic routing mechanism of the MoE layer realizes the following functions:

[0060] Assign the navigation view prediction task to the first group of expert networks;

[0061] Assign the target object recognition task to the second group of expert networks;

[0062] Only 2 expert networks are called through sparse activation, reducing the computational overhead.

[0063] (3) Construct a routing network, generate input I = T + L according to visual token T and its corresponding category information L, calculate the assignment probability of each expert network through the routing network, select the top 2 expert networks to process the visual token T, and generate the updated token T by weighted fusion of the output results ' , the specific steps are:

[0064] (a) Input I to the routing network, and get the assignment probability of each expert network through the routing network;

[0065] (b) Select the top 2 expert networks to process the visual token T;

[0066] (c) Normalize the probability values of the top two assignment probabilities through the Softmax function to get the corresponding probabilities P, which are P1 and P2, respectively;

[0067] (e) Generate the updated token T ' according to the formula T ' = P1·E1+P2·E2, where E1 and E2 are the selected expert networks.

[0068] Wherein, the visual token T includes the navigation view token T n and the target object token T o , and the corresponding category information L is the navigation view category L n and the target object category L o , respectively.

[0069] Wherein, the routing network is configured to:

[0070] Preferentially assign the navigation view token T n to the first group of expert networks that process navigation tasks;

[0071] target object wordpiece T o a second group of expert networks that are preferentially assigned to handle object recognition tasks;

[0072] only 2 expert networks are activated at a time to achieve sparse computation.

[0073] The present application verifies the feasibility of the above method through experiments, and the experiments show that, through the expert network distribution strategy of sparse activation, the model can efficiently model the heterogeneous relationship between the navigation view and the target object, while keeping a low computational overhead. Compared with the traditional multi-task learning method, the framework achieves significant improvement in view prediction accuracy, target object recognition accuracy and inference speed, effectively solving the collaborative optimization problem of language-guided navigation and pointing representation understanding tasks.

[0074] The above only describes the preferred embodiments of the present application, and it should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered within the scope of protection of the present application.

Claims

1. A method for embodied referential expression based on visual sparse hybrid experts, characterized by: The following steps are involved: Step 1: Build an end-to-end basic model, which adopts a cross-modal encoding structure based on the transformer layer, including a language instruction encoder, a panoramic observation encoder, and a cross-modal encoder; Step 2: Replace the FFN layer in the panoramic observation encoder and the cross-modal encoder with the MoE layer; Step 3: Build a routing network to dynamically assign expert networks to process different visual features based on visual word units and category information.

2. The method for embodied reference expression based on visual sparse hybrid experts according to claim 1, characterized in that: In step 1, the language instruction encoder is BERT, the panoramic encoder is a 2-layer transformer, and the cross-modal encoder is a 4-layer transformer.

3. The method for embodied reference expression based on visual sparse hybrid experts according to claim 1, characterized in that: In step 2, each MoE layer contains four expert networks, and each expert network has the same settings as the original FFN layer.

4. The method for embodied reference expression based on visual sparse hybrid experts according to claim 3, characterized in that: Among the four expert networks in each MoE layer, at least one expert network is dedicated to navigation view feature extraction, and at least another expert network is dedicated to target object feature extraction.

5. The method for embodied reference expression based on visual sparse hybrid experts according to claim 1, characterized in that: In step 3, the input of the routing network is I, which is the sum of the visual word T and the category information L represented by the word. I passes through the routing network to obtain the corresponding distribution rate of each expert network. The two expert networks with the highest distribution rate are selected to process the visual word T. The corresponding probability P is obtained by the Softmax function for the two highest distribution rates. The processing results of the two expert networks are weighted and summed according to the probability P to obtain the new visual word T. ' .

6. The method for embodied reference expression based on visual sparse hybrid experts according to claim 5, characterized in that: The visual word T includes a navigation view word T n and target object word T o , the corresponding category information L are navigation view category L n and target object category L o .

7. The method for embodied reference expression based on visual sparse hybrid experts according to claim 1, characterized in that: The dynamic routing mechanism of the MoE layer implements the following functions: Assign the navigation viewpoint prediction task to the first group of expert networks; Assign the target object recognition task to the second group of expert networks; Only two expert networks are called through sparse activation.

Images