A brain-machine fusion enhanced visual navigation method, electronic device and storage medium

Abstract

The present invention provides a brain-machine fusion visual navigation method, electronic device and storage medium, which can be applied to the field of embodied intelligent body navigation technology. The method includes: using a brain module to extract features from an observed image to obtain a neural representation with cognitive information; using a visual-target module to process target positioning information and an observed image to obtain a visual-target representation; based on a cross-attention mechanism, using a multimodal fusion module to fuse the neural representation and the visual-target representation to obtain a joint representation; using a strategy module to process the joint representation to obtain a motion control instruction, and based on the motion control instruction, controlling the target object to complete a visual navigation task. The present invention introduces advanced cognitive capabilities into a visual navigation intelligent body through neural representation, and aligns the neural representation and the visual-target representation through a multimodal fusion module to obtain a more consistent brain-vision joint representation, thereby improving the robustness of the visual navigation intelligent body.

Description

Technical Field

[0001] The present invention relates to the field of embodied intelligent body visual navigation technology, and specifically to a brain-machine fusion enhanced visual navigation method, electronic equipment and storage medium. Background Art

[0002] Embodied agent visual navigation requires the agent to use its own sensors to perceive the surrounding environment, and then use the perceived information to make decisions and output reasonable motion control instructions. At present, in the field of embodied visual navigation, the existing technical solutions have high efficiency and navigation success rate when the agent's observation is clean and disturbance-free. However, in real life, the observation of the agent is generally not clean and disturbance-free. For example, in a dimly lit scene, the observed image will be darker, and when the target object moves faster, the observed image will appear motion blurred. In addition, studies have shown that the navigation performance of the agent under perturbed observation will drop sharply, and even lead to navigation failure. Existing technical solutions use standard deep learning techniques, such as data enhancement, to improve the robustness of the agent to visual perturbations, but there is still a long way to go to achieve efficient and accurate visual navigation in real environments. Summary of the invention

[0003] In view of the above problems, the present invention provides a brain-machine fusion enhanced visual navigation method, electronic device and storage medium.

[0004] According to a first aspect of the present invention, a method for enhanced visual navigation of brain-computer fusion is provided, the method comprising:

[0005] The brain module is used to extract features from the observed image to obtain a neural representation with cognitive information, wherein the observed image is obtained by observing the target object through the trained brain-machine fusion visual navigation agent;

[0006] The target positioning information and the observed image are processed by the vision-target module to obtain the vision-target representation, wherein the target positioning information is obtained by locating the target object through the trained brain-computer fusion visual navigation agent;

[0007] Based on the cross-attention mechanism, a multimodal fusion module is used to fuse the neural representation and the visual-target representation to obtain a joint representation.

[0008] The joint representation is processed using a strategy module to obtain motion control instructions, and the target object is controlled based on the motion control instructions to complete the visual navigation task.

[0009] According to an embodiment of the present invention, the above-mentioned use of the brain module to extract features from the observed image to obtain a neural representation with cognitive information includes:

[0010] A brain module is constructed based on a deep learning model of an attention mechanism, and the brain module is pre-trained using a brain response dataset to obtain a pre-trained brain module;

[0011] Using a brain encoder of a pre-trained brain module to encode the observed image, and using a neural embedding feature extractor of the pre-trained brain module to perform neural embedding processing on the encoding result of the brain encoder to obtain a neural representation; or

[0012] Based on the encoding results of the brain encoder, a neural representation is constructed using the brain response feature extractor of the pre-trained brain module.

[0013] According to an embodiment of the present invention, the above-mentioned pre-training of the brain module using the brain response dataset to obtain the pre-trained brain module includes:

[0014] Processing the stimulus image in the brain response dataset through the encoder of the brain module to obtain a visual representation sample, wherein the labeled value of the stimulus image represents the brain response;

[0015] Using the decoder in the brain module to neurally embed the visual representation sample, and obtain a neural embedding result sample;

[0016] Use the linear prediction head of the brain module to perform data regression on the neural embedding result samples to obtain the predicted brain response samples;

[0017] Update the weight parameters of the brain module by calculating the mean square error between the predicted brain response samples and the labeled values ​​of the stimulus images;

[0018] The encoding operation, decoding operation, neural embedding operation, data regression operation and updating operation are iterated until the preset conditions are met to obtain a pre-trained brain module.

[0019] According to an embodiment of the present invention, the above-mentioned encoding process of the observed image using the brain encoder of the pre-trained brain module includes:

[0020] Using the brain encoder of the pre-trained brain module to encode the observed image to obtain an encoding result;

[0021] Decoding the encoding result using a brain decoder of a pre-trained brain module to obtain a decoding result;

[0022] The decoding results are processed by neural embedding, and the linear prediction head of the brain encoder is used to perform data regression on the neural embedding results to obtain the predicted brain response.

[0023] According to an embodiment of the present invention, the above-mentioned processing of the target positioning information and the observation image by the vision-target module to obtain the vision-target representation includes:

[0024] The visual encoder of the vision-target module is used to extract features from the observations to obtain visual features;

[0025] Compress the visual features using the compression network of the vision-target module to obtain compressed visual features;

[0026] The target encoder of the vision-target module is used to extract the target positioning information to obtain the target features;

[0027] The combined network of the vision-target module is used to concatenate the compressed visual features and target features to obtain the vision-target representation.

[0028] According to an embodiment of the present invention, the above-mentioned cross-attention mechanism is based on the multimodal fusion module to fuse the neural representation and the visual-target representation to obtain a joint representation including:

[0029] A multimodal fusion module including N computing blocks is constructed based on a cross-attention mechanism, wherein the computing block includes a brain branch and a visual branch, and N is an integer greater than or equal to 1;

[0030] The neural representation is mapped using the brain branches of each computational block to obtain multiple neural fusion representations;

[0031] The visual-target branch of each computational block is used to map the visual-target representation to obtain multiple visual-target fusion representations;

[0032] Multiple neural fusion representations and multiple visual-target representations are concatenated to obtain a brain-vision joint representation.

[0033] According to an embodiment of the present invention, the above-mentioned use of the brain branch of each computing block to map the neural representation to obtain multiple neural fusion representations includes:

[0034] Mapping the neural fusion representation obtained by the previous layer of computational blocks to a current query vector corresponding to the visual-target representation dimension using the first multilayer perceptron of the current brain branch;

[0035] The identity mapping unit of the current brain branch is used to convert the visual-target fusion representation obtained by the previous level calculation block into the current key vector and the current value vector through identity mapping;

[0036] Using the cross attention unit of the current brain branch, the current key vector, the current value vector and the current query vector corresponding to the dimension are operated to obtain the current initial neural fusion representation;

[0037] Mapping the residual sum of the current query vector and the current initial neural fusion representation into the current neural fusion representation using the second multi-layer perceptron of the current brain branch;

[0038] Multi-layer perceptron operations, identity mapping unit operations, and cross-attention unit operations are performed on each computing block to obtain multiple neural fusion representations.

[0039] According to an embodiment of the present invention, the above-mentioned strategy module includes a gated recurrent unit, an actor head and a critic head;

[0040] Among them, the gated recurrent unit is used to make the current decision based on the memory mechanism and historical decision information, the actor head is used to output the probability distribution of all actions of the current decision, and the critic head is used to output the scalar of the current decision;

[0041] Among them, the policy module is updated through the proximal policy gradient algorithm in deep reinforcement learning.

[0042] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0043] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0044] The brain-machine fusion visual navigation method provided by the present invention introduces the advanced cognitive ability of the human brain's visual system into the visual navigation agent through neural representation, and aligns the neural representation and the visual-target representation through a multimodal fusion module to obtain a more consistent brain-vision joint representation, which can improve the robustness of the visual navigation agent against various visual disturbances, and is of great significance for the deployment of embodied navigation agents in real application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0046] Figure 1 is a schematic diagram of the architecture of a brain-machine fusion robust visual navigation method according to an embodiment of the present invention;

[0047] Figure 2 is a flow chart of a visual navigation method of brain-computer fusion according to an embodiment of the present invention;

[0048] Figure 3 is a flow chart of brain module data processing according to an embodiment of the present invention;

[0049] Figure 4 is a flowchart of brain encoder pre-training according to an embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of the structure of a vision-target module according to an embodiment of the present invention;

[0051] Figure 6 is a schematic diagram of the architecture of a multimodal fusion module according to an embodiment of the present invention;

[0052] Figure 7 is a schematic diagram of the architecture of a policy module according to an embodiment of the present invention;

[0053] Figure 8 A block diagram schematically shows a structure of a visual navigation system for brain-computer fusion according to an embodiment of the present invention;

[0054] Fig. 9 A block diagram of an electronic device suitable for implementing a visual navigation method of brain-machine fusion according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0055] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0056] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0057] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0058] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0059] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0060] In response to the technical problems faced in the field of embodied visual navigation technology, the present invention provides a brain-machine fusion robustness enhanced navigation method for embodied visual navigation. Based on a standard visual navigation scheme using only visual representation, a brain encoder is first pre-trained based on a large-scale brain response dataset to simulate the human brain visual system. Then, a brain module is designed based on the brain encoder with pre-trained frozen weights to provide a neural representation with robust cognitive information of the human brain. At the same time, the visual-target module provides a visual representation. Then, the visual representation and the neural representation are jointly learned through a multimodal fusion module. Finally, a brain-visual multimodal joint representation is constructed for the state of a strategy module, so that the intelligent agent can make robust decisions in scenes with visual disturbances.

[0061] Figure 1 Schematic diagram of the architecture of the brain-computer fusion robust visual navigation method according to an embodiment of the present invention.

[0062] like Figure 1 As shown in the figure, for the visual navigation task, the observation of the agent includes two parts: the observed image and the target positioning information, which will pass through two branches: the brain module and the visual-target module. For the brain module branch, the observed image will be input into the brain module that simulates the visual cognitive system of the human brain to obtain a robust neural representation with advanced cognitive information. For the visual-target module branch, the observed image and the target positioning information will be input into the visual-target module to obtain the visual-target representation. The multimodal fusion module is used to align the neural representation and the visual-target representation, and jointly learn through the cross-attention mechanism to obtain a more consistent brain-visual joint representation. Finally, the joint representation is sent to the policy module and outputs reasonable motion control instructions. When the agent is trained, the parameters are updated through the deep reinforcement learning paradigm. Specifically, the simulation environment will receive the motion control instructions, output rewards to update the agent, and update the observation information at the same time.

[0063] In order to realize the technical solution in the above-mentioned architecture diagram, the present invention first uses a large-scale brain response dataset to pre-train the brain encoder to simulate the functions of the human brain's visual system and act as the "electronic brain" for intelligent body navigation.

[0064] Secondly, in the embodied intelligent simulator, the intelligent agent perceives the surrounding environment through the sensors it carries with it to obtain observation images and target positioning information. These environmental information will be processed through the brain branch and the visual-target branch. The brain branch is composed of brain modules, where the brain module is mainly composed of the brain encoder pre-trained in step one, and the observation image is input into the brain module to obtain a neural representation. The visual-target branch is composed of a visual-target module, which extracts the visual-target representation of the observation image and target positioning information. Then, the neural representation and the visual-target representation are jointly learned through the multimodal fusion module to obtain a brain-visual joint representation. Finally, the brain-visual joint representation is input into the policy module, and the deep reinforcement learning algorithm is used for policy learning.

[0065] The above-mentioned brain encoder refers to the software provided by the present invention which uses a Transformer-based brain encoder model to simulate the functions of the human brain visual system.

[0066] The above-mentioned brain module refers to the software provided by the present invention using a pre-trained brain encoder and a multi-layer perceptron network to construct a neural representation.

[0067] Among them, the above-mentioned vision-target module refers to the software provided by the present invention using convolutional neural networks and multi-layer perceptron networks to process the observed images and target positioning information of the intelligent agent and extract visual-target representations.

[0068] Among them, the above-mentioned multimodal fusion module refers to the software provided by the present invention, which uses a cross-attention-based network model to jointly learn neural representation and visual-target representation, and outputs brain-visual multimodal joint representation.

[0069] Among them, the above-mentioned strategy module refers to the software provided by the present invention using a strategy network based on a gated recurrent unit network and a multi-layer perceptron network to output reasonable motion control instructions according to the brain-vision multimodal joint representation.

[0070] The advantages of the visual navigation method provided by the present invention are described in more detail below through specific embodiments, flow charts and structure diagrams.

[0071] Figure 2 It is a flowchart of a visual navigation method of brain-computer fusion according to an embodiment of the present invention.

[0072] like Figure 2 As shown, the above-mentioned brain-computer fusion enhanced visual navigation method includes operations S210 to S240.

[0073] In operation S210, the brain module is used to extract features of the observed image to obtain a neural representation with cognitive information, wherein the observed image is obtained by observing the target object through the trained brain-machine fusion visual navigation agent.

[0074] In operation S220, the target positioning information and the observed image are processed by the vision-target module to obtain a vision-target representation, wherein the target positioning information is acquired by locating the target object through the trained brain-computer fusion visual navigation agent.

[0075] In operation S230, based on the cross-attention mechanism, the neural representation and the visual-target representation are fused using a multimodal fusion module to obtain a joint representation.

[0076] In operation S240, the joint representation is processed using a strategy module to obtain a motion control instruction, and the target object is controlled based on the motion control instruction to complete the visual navigation task.

[0077] The brain-machine fusion visual navigation method provided by the present invention introduces the advanced cognitive ability of the human brain's visual system into the visual navigation agent through neural representation, and aligns the neural representation and the visual-target representation through a multimodal fusion module to obtain a more consistent brain-vision joint representation, which can improve the robustness of the visual navigation agent against various visual disturbances, and is of great significance for the deployment of embodied navigation agents in real application scenarios.

[0078] The data processing flow of the brain module is further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0079] Figure 3 4 is a flow chart of brain module data processing according to an embodiment of the present invention.

[0080] like Figure 3 As shown, the above-mentioned use of brain modules to extract features of observed images to obtain neural representations with cognitive information includes: constructing brain modules based on a deep learning model of an attention mechanism, and pre-training the brain modules using a brain response dataset to obtain a pre-trained brain module; encoding the observed images using a brain encoder of the pre-trained brain module, and performing neural embedding processing on the encoding results of the brain encoder using a neural embedding feature extractor of the pre-trained brain module to obtain a neural representation; or constructing a neural representation based on the encoding results of the brain encoder using a brain response feature extractor of a pre-trained brain module.

[0081] Figure 4 is a flowchart of brain encoder pre-training according to an embodiment of the present invention.

[0082] According to an embodiment of the present invention, the above-mentioned use of a brain response data set to pre-train a brain module to obtain a pre-trained brain module includes: processing a stimulation image in the brain response data set through an encoder of the brain module to obtain a visual representation sample, wherein the label value of the stimulation image represents a brain response; using a decoder in the brain module to neurally embed the visual representation sample to obtain a neural embedding result sample; using a linear prediction head of the brain module to perform data regression on the neural embedding result sample to obtain a predicted brain response sample; updating the weight parameters of the brain module by calculating the mean square error between the predicted brain response sample and the label value of the stimulation image; iteratively performing encoding operations, decoding operations, neural embedding operations, data regression operations, and updating operations until preset conditions are met to obtain a pre-trained brain module.

[0083] In order to provide the intelligent agent with an "electronic brain" that simulates the human visual system, the present invention designs a brain encoder model based on the Transformer encoder-decoder structure and pre-trains it on a large-scale brain response dataset. Figure 4 As shown in the figure, the stimulus image first passes through the Transformer encoder to obtain a visual representation, then the visual representation passes through the Transformer decoder to obtain a neural embedding, and finally the neural embedding is regressed to predict the brain response through a linear prediction head. The weight parameters of the brain encoder are updated by calculating the mean square error between the predicted brain response and the experimentally measured brain response.

[0084] According to an embodiment of the present invention, the above-mentioned encoding processing of the observed image using the brain encoder of the pre-trained brain module includes: encoding the observed image using the brain encoder of the pre-trained brain module to obtain an encoding result; decoding the encoding result using the brain decoder of the pre-trained brain module to obtain a decoding result; performing neural embedding processing on the decoding result, and performing data regression on the neural embedding result using the linear prediction head of the brain encoder to obtain a predicted brain response.

[0085] In order to provide the intelligent agent with a neural representation with robust cognitive information of the human brain, the present invention designs a brain module, which is composed of a brain encoder with pre-trained frozen weights, a neural embedding feature extractor, and a brain response feature extractor. The neural embedding feature extractor is composed of a multi-layer perceptron model, which processes the neural embedding output by the Transformer decoder in the brain encoder and outputs a neural representation. Another method is to construct a neural representation through a brain response feature extractor, which processes the predicted brain response output by the linear prediction head in the brain encoder and outputs a neural representation. The two neural representations have different biological properties, and you can choose one of them when using it.

[0086] According to an embodiment of the present invention, the above-mentioned use of the vision-target module to process the target positioning information and the observed image to obtain the visual-target representation includes: using the visual encoder of the vision-target module to extract features of the observation to obtain visual features; using the compression network of the vision-target module to compress the visual features to obtain compressed visual features; using the target encoder of the vision-target module to extract features of the target positioning information to obtain target features; using the combined network of the vision-target module to splice the compressed visual features and the target features to obtain the visual-target representation.

[0087] The data processing flow of the vision-target module is further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0088] Figure 5 is a schematic diagram of the structure of a vision-target module according to an embodiment of the present invention.

[0089] like Figure 5 As shown, at each time step, the agent can obtain the observed image and target positioning information. In order to extract their visual-target features, the present invention designs a visual-target module, which is composed of a visual encoder, a target encoder, a compression network and a combined network. The visual encoder uses ResNet-18 pre-trained on ImageNet to extract the visual features of the observed image, and then the compression network processes the high-dimensional visual features to obtain compressed visual features. At the same time, the target encoder extracts the target features of the target positioning information, and then the visual features and target features are spliced ​​and the visual-target representation is obtained through the combined network.

[0090] According to an embodiment of the present invention, the above-mentioned cross-attention mechanism-based, multimodal fusion module is used to fuse the neural representation and the visual-target representation to obtain a joint representation, including: constructing a multimodal fusion module including N computing blocks based on the cross-attention mechanism, wherein the computing block includes a brain branch and a visual branch, and N is an integer greater than or equal to 1; using the brain branch of each computing block to map the neural representation to obtain multiple neural fusion representations; using the visual-target branch of each computing block to map the visual-target representation to obtain multiple visual-target fusion representations; multiple neural fusion representations and multiple visual-target representations are spliced ​​to obtain a brain-visual joint representation.

[0091] According to an embodiment of the present invention, the above-mentioned use of the brain branch of each computing block to map the neural representation to obtain multiple neural fusion representations includes: using the first multilayer perceptron of the current brain branch to map the neural fusion representation obtained by the previous layer computing block to a current query vector corresponding to the visual-target representation dimension; using the identity mapping unit of the current brain branch to convert the visual-target fusion representation obtained by the previous layer computing block into a current key vector and a current value vector through identity mapping; using the cross-attention unit of the current brain branch to operate the current key vector, current value vector and current query vector corresponding to the dimension to obtain the current initial neural fusion representation; using the second multilayer perceptron of the current brain branch to map the residual sum of the current query vector and the current initial neural fusion representation to the current neural fusion representation; performing multilayer perceptron operations, identity mapping unit operations, and cross-attention unit operations on each computing block to obtain multiple neural fusion representations.

[0092] The processing flow of the visual-target branch is similar to that of the brain branch. Technicians in the relevant technical field can use the visual-target branch to process the visual-target representation under the inspiration of the processing flow of the brain branch.

[0093] The data processing process of the multimodal fusion module is further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0094] Figure 6 is a schematic diagram of the architecture of a multimodal fusion module according to an embodiment of the present invention.

[0095] like Figure 6 The data processing flow of the multimodal fusion module shown in Figure 2 is the neural representation and visual-target representation The present invention designs a multimodal fusion module based on cross-attention in order to jointly learn the representations of these two different modalities. The multimodal fusion module consists of N stacked computational blocks and includes brain branches and visual branches. For the brain branch part of the i-th computational block, the neural representation First, the multi-layer perceptron MLP is used to map the query vector corresponding to the visual-target representation dimension. Then, the visual-object representation is converted into a key vector via identity mapping. and Value vector Next, the query vector, key vector, and value vector enter the cross-attention module for operation to obtain the feature Finally, the multi-layer perceptron MLP is used to transform and The residual and mapping are features For the symmetric visual branch, similar operations are performed, except that the sources of the query vector, key vector, and value vector are different, and the final feature is obtained. The fused neural representation after N computational blocks and fused visual representations Concatenate to get the final brain-vision joint representation .

[0096] According to an embodiment of the present invention, the above-mentioned strategy module includes a gated recurrent unit, an actor head and a critic head; wherein the gated recurrent unit is used to make current decisions based on a memory mechanism and historical decision information, the actor head is used to output the probability distribution of all actions of the current decision, and the critic head is used to output the scalar of the current decision; wherein the strategy module is updated by the proximal policy gradient algorithm in deep reinforcement learning.

[0097] Figure 7 4 is a schematic diagram of the architecture of a policy module according to an embodiment of the present invention.

[0098] like Figure 7 As shown in the figure, the policy module consists of three parts: a gated recurrent unit, an actor head, and a critic head. First, the brain-vision joint representation is fed into the gated recurrent unit, which uses the memory mechanism to make better current decisions using historical decision information. In addition, the actor head and the critic head are composed of a fully connected network, where the actor head outputs the probability distribution of the decision action and the critic head outputs a scalar. The entire policy module is updated using the proximal policy gradient algorithm in deep reinforcement learning.

[0099] Figure 8 The structural block diagram of the brain-computer fusion visual navigation system according to an embodiment of the present invention is schematically shown.

[0100] like Figure 8 As shown, the brain-machine fusion visual navigation system 800 of this embodiment includes a neural representation acquisition module 810, a visual-target representation acquisition module 820, a joint representation acquisition module 830 and a visual navigation control module 840.

[0101] The neural representation acquisition module 810 is used to extract features from the observed image using the brain module to obtain a neural representation with cognitive information, wherein the observed image is observed and acquired by the trained brain-machine fusion visual navigation agent for the target object. In one embodiment, the neural representation acquisition module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0102] The visual-target representation acquisition module 820 is used to process the target positioning information and the observed image using the visual-target module to obtain the visual-target representation, wherein the target positioning information is obtained by positioning the target object through the trained brain-computer fusion visual navigation agent. In one embodiment, the visual-target representation acquisition module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0103] The joint representation acquisition module 830 is used to fuse the neural representation and the visual-target representation based on the cross attention mechanism using the multimodal fusion module to obtain a joint representation. In one embodiment, the joint representation acquisition module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0104] The visual navigation control module 840 is used to process the joint representation using the strategy module to obtain motion control instructions, and control the target object based on the motion control instructions to complete the visual navigation task. In one embodiment, the visual navigation control module 840 can be used to perform the operation S230 described above, which will not be repeated here.

[0105] According to an embodiment of the present invention, any multiple modules of the neural representation acquisition module 810, the visual-target representation acquisition module 820, the joint representation acquisition module 830, and the visual navigation control module 840 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the neural representation acquisition module 810, the visual-target representation acquisition module 820, the joint representation acquisition module 830, and the visual navigation control module 840 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the neural representation acquisition module 810, the visual-target representation acquisition module 820, the joint representation acquisition module 830, and the visual navigation control module 840 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0106] According to an embodiment of the present invention, the neural representation acquisition module 810 includes a pre-training module, a neural embedding module or a neural construction module.

[0107] Among them, the pre-training module is used to build a brain module based on a deep learning model of the attention mechanism, and use the brain response dataset to pre-train the brain module to obtain a pre-trained brain module.

[0108] A neural embedding module, used to encode the observed image using a brain encoder of a pre-trained brain module, and to perform neural embedding processing on the encoding result of the brain encoder using a neural embedding feature extractor of the pre-trained brain module to obtain a neural representation; or

[0109] The neural construction module is used to construct a neural representation based on the encoding results of the brain encoder using the brain response feature extractor of the pre-trained brain module.

[0110] According to an embodiment of the present invention, the pre-training module includes an encoding module, a decoding and embedding module, a data regression module, an updating module and an iterative training module.

[0111] The encoding module is used to process the stimulus image in the brain response data set through the encoder of the brain module to obtain a visual representation sample, wherein the label value of the stimulus image represents the brain response.

[0112] A decoding and embedding module, used for performing neural embedding on the visual representation sample using the decoder in the brain module to obtain a neural embedding result sample;

[0113] A data regression module, used to perform data regression on the neural embedding result samples using the linear prediction head of the brain module to obtain predicted brain response samples;

[0114] An updating module, used for updating the weight parameters of the brain module by calculating the mean square error between the predicted brain response samples and the labeled values ​​of the stimulus images;

[0115] The iterative training module is used to iteratively perform encoding operations, decoding operations, neural embedding operations, data regression operations, and update operations until preset conditions are met to obtain a pre-trained brain module.

[0116] According to an embodiment of the present invention, the above-mentioned neural embedding module includes an encoding unit, a decoding unit and a neural embedding and data regression unit.

[0117] The encoding unit is used to encode the observed image using the brain encoder of the pre-trained brain module to obtain an encoding result.

[0118] A decoding unit, used for decoding the encoding result using a brain decoder of a pre-trained brain module to obtain a decoding result;

[0119] The neural embedding and data regression unit is used to perform neural embedding processing on the decoding results, and use the linear prediction head of the brain encoder to perform data regression on the neural embedding results to obtain the predicted brain response.

[0120] According to an embodiment of the present invention, the vision-target module 820 includes a visual feature extraction module, a feature compression module, a positioning information feature extraction module and a feature splicing module.

[0121] The visual feature extraction module is used to extract features from observations using the visual encoder of the vision-target module to obtain visual features.

[0122] The feature compression module is used to compress the visual features using the compression network of the vision-target module to obtain compressed visual features.

[0123] The positioning information feature extraction module is used to extract features of the target positioning information using the target encoder of the vision-target module to obtain target features.

[0124] The feature concatenation module uses the combined network of the vision-target module to concatenate the compressed visual features and target features to obtain the vision-target representation.

[0125] According to an embodiment of the present invention, the multimodal fusion module 830 includes a model building module, a neural representation mapping module, a visual-target representation mapping module and a multimodal splicing module.

[0126] A model building module is used to build a multimodal fusion module including N computing blocks based on a cross-attention mechanism, wherein the computing block includes a brain branch and a visual branch, and N is an integer greater than or equal to 1.

[0127] The neural representation mapping module is used to map the neural representation using the brain branches of each computing block to obtain multiple neural fusion representations.

[0128] The vision-target representation mapping module is used to map the vision-target representation using the vision-target path of each computing block to obtain multiple vision-target fusion representations.

[0129] The multimodal splicing module is used to splice multiple neural fusion representations and multiple visual-target representations to obtain a brain-vision joint representation.

[0130] According to an embodiment of the present invention, the neural representation mapping module includes a first mapping module, an identity mapping module, a vector operation module, a second mapping module and an iterative operation module.

[0131] The first mapping module is used to map the neural fusion representation obtained by the previous layer computing block into a current query vector corresponding to the visual-target representation dimension using the first multi-layer perceptron of the current brain branch.

[0132] The identity mapping module is used to use the identity mapping unit of the current brain branch to convert the visual-target fusion representation obtained by the previous level calculation block into the current key vector and the current value vector through identity mapping.

[0133] A vector operation module is used to operate the current key vector, the current value vector and the current query vector corresponding to the dimension by using the cross attention unit of the current brain branch to obtain the current initial neural fusion representation;

[0134] The second mapping module is used to map the residual sum of the current query vector and the current initial neural fusion representation into the current neural fusion representation by using the second multi-layer perceptron of the current brain branch.

[0135] The iterative operation module is used to perform multi-layer perceptron operations, identity mapping unit operations, and cross-attention unit operations on each calculation block to obtain multiple neural fusion representations.

[0136] Fig. 9 A block diagram of an electronic device suitable for implementing a visual navigation method of brain-machine fusion according to an embodiment of the present invention is schematically shown.

[0137] like Fig. 9 As shown, the electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 to a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0138] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0139] According to an embodiment of the present invention, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the input / output (I / O) interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0140] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0141] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0144] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A brain-computer fusion enhanced visual navigation method, characterized in that: The method comprises: Using the brain module to extract features from the observed image to obtain a neural representation with cognitive information, wherein the observed image is obtained by observing the target object through the trained brain-machine fusion visual navigation agent; The target positioning information and the observed image are processed by a vision-target module to obtain a vision-target representation, wherein the target positioning information is obtained by locating the target object through a trained brain-computer fusion visual navigation agent; Based on the cross-attention mechanism, a multimodal fusion module is used to fuse the neural representation and the visual-target representation to obtain a joint representation; Processing the joint representation using a strategy module to obtain a motion control instruction, and controlling the target object to complete a visual navigation task based on the motion control instruction; Wherein, based on the cross attention mechanism, the neural representation and the visual-target representation are fused using a multimodal fusion module to obtain a joint representation including: A multimodal fusion module including N computing blocks is constructed based on a cross-attention mechanism, wherein the computing block includes a brain branch and a visual branch, and N is an integer greater than or equal to 1; Using the brain branches of each of the computing blocks to perform mapping processing on the neural representation to obtain multiple neural fusion representations; Using the vision-target branch of each of the computing blocks to perform mapping processing on the vision-target representation to obtain multiple vision-target fusion representations; The multiple neural fusion representations and the multiple visual-target representations are spliced ​​together to obtain a brain-vision joint representation.

2. The method according to claim 1, characterized in that The brain module is used to extract features from the observed image, and the neural representations with cognitive information are obtained, including: Building a brain module based on a deep learning model of an attention mechanism, and pre-training the brain module using a brain response dataset to obtain a pre-trained brain module; Using the brain encoder of the pre-trained brain module to encode the observed image, and using the neural embedding feature extractor of the pre-trained brain module to perform neural embedding processing on the encoding result of the brain encoder to obtain the neural representation; or Based on the encoding result of the brain encoder, the neural representation is constructed using the brain response feature extractor of the pre-trained brain module.

3. The method according to claim 2, characterized in that The brain module is pre-trained using the brain response data set, and the pre-trained brain module includes: Processing the stimulus image in the brain response data set through the encoder of the brain module to obtain a visual representation sample, wherein the label value of the stimulus image represents the brain response; Using the decoder in the brain module to perform neural embedding on the visual representation sample to obtain a neural embedding result sample; Using the linear prediction head of the brain module to perform data regression on the neural embedding result samples to obtain predicted brain response samples; updating the weight parameters of the brain module by calculating the mean square error between the predicted brain response samples and the labeled values ​​of the stimulus image; The encoding operation, decoding operation, neural embedding operation, data regression operation and updating operation are iterated until the preset conditions are met to obtain a pre-trained brain module.

4. The method according to claim 2, characterized in that: The process of encoding the observed image using the brain encoder of the pre-trained brain module includes: Using the brain encoder of the pre-trained brain module to encode the observed image to obtain an encoding result; Decoding the encoding result using the brain decoder of the pre-trained brain module to obtain a decoding result; The decoding result is subjected to neural embedding processing, and the linear prediction head of the brain encoder is used to perform data regression on the neural embedding result to obtain a predicted brain response.

5. The method according to claim 1, characterized in that The target positioning information and the observed image are processed by the vision-target module to obtain a vision-target representation including: Extracting features from the observations using a visual encoder of the vision-target module to obtain visual features; Compressing the visual features using the compression network of the vision-object module to obtain compressed visual features; Using the target encoder of the vision-target module to extract features from the target positioning information to obtain target features; The compressed visual features and the target features are concatenated using a combined network of the visual-target module to obtain the visual-target representation.

6. The method according to claim 1, characterized in that The neural representation is mapped using the brain branch of each of the computing blocks to obtain a plurality of neural fusion representations including: Mapping the neural fusion representation obtained by the previous level computing block to a current query vector corresponding to the visual-target representation dimension using the first multilayer perceptron of the current brain branch; The identity mapping unit of the current brain branch is used to convert the visual-target fusion representation obtained by the previous level computing block into a current key vector and a current value vector through identity mapping; Using the cross attention unit of the current brain branch to operate the current key vector, the current value vector and the current query vector corresponding to the dimension to obtain the current initial neural fusion representation; Mapping the residual sum of the current query vector and the current initial neural fusion representation into a current neural fusion representation using a second multilayer perceptron of the current brain branch; A multi-layer perceptron operation, an identity mapping unit operation, and a cross-attention unit operation are performed on each of the computing blocks to obtain a plurality of the neural fusion representations.

7. The method according to claim 1, characterized in that The strategy module includes a gated recurrent unit, an actor head, and a critic head; The gated recurrent unit is used to make a current decision based on a memory mechanism and historical decision information, the actor head is used to output the probability distribution of all actions of the current decision, and the critic head is used to output the scalar of the current decision; Among them, the policy module is updated through the proximal policy gradient algorithm in deep reinforcement learning.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program or instruction is stored thereon, wherein when the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images