A method and related device for autonomous driving visual reasoning cognition

Abstract

The present invention discloses a method for visual reasoning and cognition of autonomous driving and a related device. The present invention adopts a human brain visual pathway model that imitates the visual pathway to perform visual reasoning and cognition of autonomous driving, and uses a neuron model to simulate the process of visual signals entering from the retina to the lateral geniculate body and then to the primary visual cortex, which is conducive to extracting the underlying features of the image and laying the foundation for the subsequent semantic information cognition. The neuron model is used to simulate the ventral pathway and dorsal pathway of the visual pathway, which is conducive to high-level cognition of feature maps and question texts and assists in understanding the logic of image composition. The neuron model is used to simulate the prefrontal lobe of the human brain, which is conducive to inferring the answers corresponding to the image and question text, and realizes visual reasoning based on the visual features and semantic cognition of the visual pathway, fully opens up the neural cognitive process, and improves interpretability.

Description

Technical Field

[0001] The present invention relates to an autonomous driving visual reasoning cognition method and related devices, and belongs to the field of visual reasoning. Background Art

[0002] In the autonomous driving scenario, by understanding and inferring the images captured by the camera, the vehicle can dynamically perceive and respond to complex traffic environments in real time. It can not only identify and locate objects such as pedestrians, vehicles, and traffic signs, but also analyze the scene and predict the behavior of dynamic objects, such as determining whether pedestrians intend to cross the road, thereby making safe and intelligent driving decisions.

[0003] Visual reasoning technology ensures that autonomous driving systems can operate efficiently and accurately in a variety of complex and dynamic traffic scenarios, improving driving safety and reliability. The development of visual reasoning tasks is inseparable from the construction and disclosure of large-scale datasets, which contain a large number of images, image-corresponding questions and answer pairs, providing a standardized benchmark for evaluating the performance of different models.

[0004] The main challenge of visual reasoning is to integrate multimodal information. In addition, how the model can perform complex reasoning on the graph based on understanding multimodal features and combining specific problems is another important challenge. In response to these challenges, the realization of visual reasoning cognition can be summarized into three tasks: 1. How to identify the visual features of an image; 2. How to understand the semantic information of an image and the relationship between objects.

[0005] For the first task, convolutional neural networks are currently commonly used to process image features, but this method lacks biological interpretability; for the second task, multimodal fusion methods are currently commonly used, such as attention mechanisms, bilinear pooling, and graph neural networks, but this mechanism cannot handle complex reasoning problems and lacks biological interpretability. Therefore, the existing methods lack biological interpretability, resulting in low computational efficiency and high energy consumption. Summary of the invention

[0006] The present invention provides an autonomous driving visual reasoning cognition method and related devices, which solve the problems disclosed in the background technology.

[0007] According to one aspect of the present disclosure, a method for autonomous driving visual reasoning cognition is provided, comprising:

[0008] Obtain images in autonomous driving scenarios and corresponding question texts;

[0009] According to the image and the corresponding question text, the human brain visual pathway model is used to obtain the answer corresponding to the image and the question text;

[0010] Among them, the working process of the human brain visual pathway model is:

[0011] The retina model, lateral geniculate body model, visual cortex V1 model and visual cortex V2 model are used to perform low-level and intermediate-level cognition on the image and obtain images after diversity encoding.

[0012] Perform feature extraction on the diversity-encoded image, and obtain a feature map of each object in the image based on the extracted features;

[0013] The visual cortex V3 model, visual cortex V4 model, visual cortex V5 model, intraparietal sulcus model and inferior temporal cortex model are used to perform high-level cognition on the feature map and question text to obtain the final cognitive result of each object in the image;

[0014] The prefrontal lobe model is used to infer the final cognitive results and obtain the answers corresponding to the image and question text;

[0015] Each model in the human brain visual pathway model is used to imitate each cerebral cortex of the visual pathway, and a neuron model is used to model cells of the cerebral cortex.

[0016] In some embodiments of the present disclosure, the human brain visual pathway model further includes a priori alignment network;

[0017] Before performing low-level and mid-level cognition on the image, a prior alignment network is used to align the image and question text a priori.

[0018] In some embodiments of the present disclosure, a retinal model, a lateral geniculate body model, a visual cortex V1 model, and a visual cortex V2 model are used to perform low-level and intermediate cognition on an image to obtain a diversity-encoded image, including:

[0019] The retina model is used to convert images into pulse signals;

[0020] The lateral geniculate body model was used to integrate the impulse signals;

[0021] The visual cortex V1 model is used to extract multi-channel features of the output signal of the lateral geniculate body model, and the features of the output of different pathways are encoded;

[0022] The visual cortex V2 model is used to further extract the encoded features and obtain diverse encoded images.

[0023] In some embodiments of the present disclosure, in the visual cortex V1 model, when receiving the output of the lateral geniculate body model, a feedback pulse is sent to the lateral geniculate body model to adjust the firing rate of the output layer of the lateral geniculate body model.

[0024] In some embodiments of the present disclosure, a visual cortex V3 model, a visual cortex V4 model, a visual cortex V5 model, an intraparietal sulcus model, and an inferior temporal cortex model are used to perform high-level cognition on feature maps and question texts to obtain the final cognitive results of each object in the image, including:

[0025] The visual cortex V4 model and the visual cortex V3 model are used to extract the object nodes and the relationship between the object nodes in the diversity-encoded image, and a scene graph is generated according to the relationship between the object nodes and the object nodes.

[0026] The visual cortex V5 model is used to parse the action of the question text, and the intraparietal sulcus model is used to plan the action parsing results. According to the planning results and the scene graph, a directed sequence of the scene graph is generated;

[0027] According to the directed sequence of the scene graph, the inferior temporal cortex model is used to perform walk-back causation on the scene graph to obtain the final cognitive result of each object in the image.

[0028] According to another aspect of the present disclosure, there is provided an autonomous driving visual reasoning cognition device, comprising:

[0029] An acquisition module is used to obtain images in autonomous driving scenarios and corresponding question texts.

[0030] The cognitive module uses the human brain visual pathway model to obtain the answer corresponding to the image and the question text based on the image and the question text;

[0031] Among them, the working process of the human brain visual pathway model is:

[0032] The retina model, lateral geniculate body model, visual cortex V1 model and visual cortex V2 model are used to perform low-level and intermediate-level cognition on the image and obtain images after diversity encoding.

[0033] Perform feature extraction on the diversity-encoded image, and obtain a feature map of each object in the image based on the extracted features;

[0034] The visual cortex V3 model, visual cortex V4 model, visual cortex V5 model, intraparietal sulcus model and inferior temporal cortex model are used to perform high-level cognition on the feature map and question text to obtain the final cognitive result of each object in the image;

[0035] The prefrontal lobe model is used to infer the final cognitive results and obtain the answers corresponding to the image and question text;

[0036] Each model in the human brain visual pathway model is used to imitate each cerebral cortex of the visual pathway, and a neuron model is used to model cells of the cerebral cortex.

[0037] In some embodiments of the present disclosure, in the human brain visual pathway model of the cognitive module, a retinal model, a lateral geniculate body model, a visual cortex V1 model, and a visual cortex V2 model are used to perform low-level and intermediate cognition on the image to obtain a diversely encoded image, including:

[0038] The retina model is used to convert images into pulse signals;

[0039] The lateral geniculate body model was used to integrate the impulse signals;

[0040] The visual cortex V1 model is used to extract multi-channel features of the output signal of the lateral geniculate body model, and the features of the output of different pathways are encoded;

[0041] The visual cortex V2 model is used to further extract the encoded features and obtain diverse encoded images.

[0042] In some embodiments of the present disclosure, in the human brain visual pathway model of the cognitive module, the visual cortex V3 model, the visual cortex V4 model, the visual cortex V5 model, the intraparietal sulcus model and the inferior temporal cortex model are used to perform high-level cognition on the feature map and the question text to obtain the final cognitive result of each object in the image, including:

[0043] The visual cortex V4 model and the visual cortex V3 model are used to extract the object nodes and the relationship between the object nodes in the diversity-encoded image, and a scene graph is generated according to the relationship between the object nodes and the object nodes.

[0044] The visual cortex V5 model is used to parse the action of the question text, and the intraparietal sulcus model is used to plan the action parsing results. According to the planning results and the scene graph, a directed sequence of the scene graph is generated;

[0045] According to the directed sequence of the scene graph, the inferior temporal cortex model is used to perform walk-back causation on the scene graph to obtain the final cognitive result of each object in the image.

[0046] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device performs an autonomous driving visual reasoning cognition method.

[0047] According to another aspect of the present disclosure, a computer device is provided, comprising one or more processors and one or more memories, wherein one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing an autonomous driving visual reasoning cognition method.

[0048] The beneficial effects achieved by the present invention are as follows: the present invention adopts a human brain visual pathway model that imitates the visual pathway to perform visual reasoning and cognition in autonomous driving, and uses a neuron model to simulate the process of visual signals entering from the retina to the lateral geniculate body and then to the primary visual cortex, which is conducive to extracting the underlying features of the image and laying the foundation for subsequent semantic information cognition. The neuron model is used to simulate the ventral pathway and dorsal pathway of the visual pathway, which is conducive to high-level cognition of feature maps and question texts and assists in understanding the logic of image composition. The neuron model is used to simulate the prefrontal lobe of the human brain, which is conducive to inferring the answers corresponding to the image and question text, and realizes visual reasoning based on the visual features and semantic cognition of the visual pathway, fully opens up the neural cognitive process, and improves interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of the visual reasoning cognitive method for autonomous driving;

[0050] Figure 2 It is a priori aligned network diagram in the human brain visual pathway model;

[0051] Figure 3 This is a model diagram corresponding to low-level and intermediate cognition in the human brain visual pathway model;

[0052] Figure 4 This is a model diagram corresponding to high-level cognition in the human brain visual pathway model;

[0053] Figure 5 This is a diagram of the prefrontal lobe model in the human brain visual pathway model;

[0054] Figure 6 Block diagram of the visual reasoning cognitive device for autonomous driving. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0056] Unless otherwise specified, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0057] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0058] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0059] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0060] It should be noted that similar symbols and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it does not require further discussion in subsequent figures.

[0061] In order to solve the problem that existing autonomous driving visual reasoning and cognition methods lack biological explainability, the present disclosure proposes a new autonomous driving visual reasoning and cognition method and related devices, specifically, using a human brain visual pathway model that imitates the visual pathway to perform autonomous driving visual reasoning and cognition.

[0062] Figure 1 This is a schematic diagram of an embodiment of the autonomous driving visual reasoning cognition method disclosed in the present invention. Figure 1 Embodiments of may be executed by a controller of an autonomous driving system.

[0063] like Figure 1 As shown, step 1 of the embodiment is to obtain an image in an autonomous driving scenario and a question text corresponding to the image.

[0064] It should be noted that the image is captured by the camera in the autonomous driving scenario, and the question is the problem of human input corresponding to the image; for example: the image is a road photo taken by a camera at an intersection, and the question is how many blue cars are there on the left.

[0065] return Figure 1 In step 1 of the embodiment, based on the image and the question text corresponding to the image, a human brain visual pathway model is used to obtain the answer corresponding to the image and the question text.

[0066] It should be noted that the human brain visual pathway model is used to simulate the visual pathway, and includes multiple models internally, each model simulates each cerebral cortex of the visual pathway, and uses a neuron model to model cells of the cerebral cortex.

[0067] In some embodiments, the human brain visual pathway model will first use a priori alignment network to perform a priori alignment on the image and question text, which can improve the accuracy of subsequent models and reduce training time.

[0068] It should be noted that see Figure 2The prior alignment network mainly includes image encoding, text encoding, joint encoding, etc.; among them, the joint encoding part adopts CLIP (1024 dimensions) trained by OpenAI (Contrastive Language-Image Pre-Training, language-image contrast pre-training model), the image encoding part adopts ResNet50, and the text encoding part adopts Transformer (deep neural network model based on self-attention mechanism).

[0069] The alignment process of the prior alignment network can be described as: using ResNet50 to encode the image, using Transformer to encode the question text, and using CLIP to jointly encode the outputs of ResNet50 and Transformer to obtain the result of joint encoding; specifically, the visual V is aligned with the question Q, and after alignment, the image is also segmented (i.e., v 1. v 2. v 3) Input the subsequent model.

[0070] It should be noted that the loss function of the prior alignment network is determined by two parts: 1. The cosine similarity between the image and the text description; 2. The brain voxel prediction loss, measured by the Pearson correlation coefficient; it can be expressed as:

[0071] ;

[0072] ;

[0073] ;

[0074] In the formula, L sim is the familiarity loss, sim represents the cosine similarity, N 1 is the number of image-text pairs calculated, I Represents a collection, Indicates A question, Indicates images, represents any image in the collection, is a constant, L Brain is the brain voxel prediction loss, calculated by the ridge regression model, R 2 ( r true , r pred ) is the true value r trueand predicted values r pred The Pearson correlation coefficient between L is the total loss, λ is a hyperparameter.

[0075] After the image and question text are aligned a priori, the retina model, lateral geniculate body model, visual cortex V1 model and visual cortex V2 model are used to perform low-level and intermediate cognition on the image to obtain a diversely encoded image, see Figure 3 .

[0076] It should be noted that in the human visual pathway, the image starts from the retina and passes through the lateral geniculate nucleus (LGN), visual cortex V1, and visual cortex V2 in sequence. This part is the low-level and intermediate processing process of the human visual pathway. By imitating this process, we can construct a retina model, a lateral geniculate nucleus model, a visual cortex V1 model, and a visual cortex V2 model, thereby forming a diverse encoding of the image.

[0077] The retina model is used to convert the image into a pulse signal, the lateral geniculate body model is used to integrate the pulse signals, the visual cortex V1 model is used to perform multi-channel feature extraction on the output signal of the lateral geniculate body model, and the features of the output of different pathways are encoded. The visual cortex V2 model is used to further extract the encoded features to obtain a diversely encoded image.

[0078] It should be noted that the retina is structurally divided into three layers: photoreceptor layer, bipolar cell layer and ganglion cell layer. The neurons in the photoreceptor layer are simulated using the first LIF (Leaky Integrate-and-Fire) model, the neurons in the bipolar cell layer are simulated using the second LIF model, and the neurons in the ganglion cell layer are simulated using the third LIF model.

[0079] In the retinal model, the photoreceptor cell model, the bipolar cell model and the ganglion cell model are connected in sequence, and the cell models are specifically connected through the synapse model.

[0080] The rod bipolar neuron cell model and the pyramidal ganglion cell model are connected by an inhibitory synapse model to regulate the occurrence rate; the models of the same type of on or off pyramidal bipolar neuron cells and pyramidal ganglion cells are connected by an excitatory synapse model, and the models of the different types of on or off pyramidal bipolar neuron cells and pyramidal ganglion cells are connected by an inhibitory synapse model; multiple presynaptic neuron cell models are connected to one postsynaptic neuron cell model, and the weight initialization of the presynaptic neuron cell model and the postsynaptic neuron cell model follows a Gaussian distribution; in the ganglion cell layer, excitatory connections are used between central neurons and inhibitory connections are used between surrounding neurons.

[0081] It should be noted that the LIF model can be expressed as:

[0082] ;

[0083] In the formula, V p ( t ) is the neuron cell potential voltage, I in ( t ) is the LIF model input current, R m is the equivalent membrane resistance, V rest is the resting potential, t Indicates time, No. j The time of a presynaptic spike, V th is the threshold potential, S ( t ) is the output pulse of the LIF model.

[0084] It should be noted that the first LIF model is an improved LIF model, specifically the output is improved, and its output is membrane potential, which is more in line with biological explanation and can describe signals more accurately and diversely. The outputs of other LIF models are pulse signals, that is, 0 or 1.

[0085] It should be noted that neurons communicate with each other through synapses. Different presynaptic neurons are simulated by different synaptic models. For presynaptic neurons using the LIF model, the synaptic model can be expressed as:

[0086] ;

[0087] In the formula, For the i The input current of the postsynaptic neuron cell model, P ( i ) is connected to thei A collection of presynaptic neuron cell models and postsynaptic neuron cell models, S j ( t ) is the j The output pulse of the presynaptic neuron cell model, w ji ( t ) is the i The postsynaptic neuron cell model and the j The synaptic weights between the presynaptic neuron cell models, if the excitatory synapse is connected, then w ji ( t ) is greater than 0, if the inhibitory synapse is connected, then w ji ( t ) is less than 0.

[0088] It should be noted that the STDP (Spike Timing Dependent Plasticity) learning rule is implemented on the synapse to extract features and simulate the learning rule in the primary visual pathway. The change of synaptic weight is related to the change of the time difference between spikes. The formula can be expressed as:

[0089] ;

[0090] In the formula, w is the synaptic weight, Δ w is the change in synaptic weight, They are Δ t >0 and Δ t <0 constant;

[0091] ;

[0092] In the formula, F + ( w )and F - ( w ) is the defined learning function, λ 1 is the learning rate, α To provide F + and F - The coefficient of asymmetry between t post and t pre are the spike times of the presynaptic and postsynaptic neurons, Δ t for tpost and t pre The difference.

[0093] According to the parameters μ The STDP learning rules for the value of 1 are divided into two types: one is μ 1=0, called additive STDP, the other is μ 1=1, which is called multiplicative STDP; here we use additive STDP to simplify the operation. In addition, for the matching of pulses, the nearest neighbor scheme is used to reduce the consumption of computing resources. This scheme can be described as: when a presynaptic pulse is generated, only the nearest postsynaptic pulse is matched.

[0094] It should be noted that the lateral geniculate body includes three types of cells, namely P cells, M cell layer and K cells. These cells are simulated using the LIF model, that is, defined as the P cell model, the M cell layer model and the K cell model. The parallel P cell model, the M cell layer model and the K cell model constitute the lateral geniculate body model. The P cell model, the M cell layer model and the K cell model all receive the output of the retinal model and forward the output of the retinal model to the visual cortex V1 model.

[0095] Figure 3 It can be seen that the lateral geniculate body model is actually a transit model, which can be simplified into an input layer and an output layer. The input layer includes three channels, P, M, and K, which have a one-to-one excitatory connection with the ganglion on, and an inhibitory connection with the ganglion off with a receptive field radius of 1; the output layer is also divided into three channels, P, M, and K, which have a one-to-one excitatory connection with the input.

[0096] It should be noted that the visual cortex V1 has a total of six layers in structure. Considering the cognitive function of the visual pathway, it can be functionally divided into three layers, namely the L6 simple cell layer, the L4 simple cell layer and the L2 / 3 complex cell layer. Therefore, the visual cortex V1 model can include the L6 simple cell model, the L4 simple cell model and the L2 / 3 complex cell model, all of which use the LIF model.

[0097] Among them, when the L6 simple cell model receives the output of the P, M, and K channels of the lateral geniculate body model, it sends feedback pulses to the lateral geniculate body model to adjust the firing rate of the output layer of the lateral geniculate body model, thereby improving the accuracy of the features; the L4 simple cell model receives the output of the lateral geniculate body model and uses the shape channel, color channel, and position channel to extract features from the received output; the L2 / 3 complex cell model encodes the features of the outputs of different pathways. The synapses between the L4 simple cell model and the L2 / 3 complex cell model are all excitatory synapses (excitatory synapses represent positive weights), and each postsynaptic neuron cell has a receptive field with a radius of 1.

[0098] Figure 3 In the model, there are three color channels, and the neurons in the color channels have a pair of receptive fields (i.e., the field of view of the neurons, the range of images that can be processed). One color channel receives the output of the M cell layer model, one color channel receives the output of the P cell model, and one color channel receives the joint output of the P cell model, the M cell layer model, and the K cell model. Each color channel extracts red, yellow, and blue information respectively. The three color channels are stacked to form the color channel of the visual cortex V1 model.

[0099] There are eight shape channels, each of which inputs eight orientation features. The postsynaptic neuron cells in each channel have a receptive field similar to that of a Gabor filter, extracting edge orientation features at different scales and preferred directions. The eight shape channels are stacked to form the shape channel of the visual cortex V1 model.

[0100] There are two position channels, each of which inputs the horizontal and vertical coordinates. The postsynaptic neuron cells in each channel have a receptive field similar to that of a Gabor filter, which extracts position features in the horizontal and vertical directions respectively. The two position channels are stacked to form the position channel of the visual cortex V1 model.

[0101] It should be noted that the visual cortex V2 model adopts the LIF model, which acts as a fully connected layer to further extract the features output by the visual cortex V1 model.

[0102] After diversity encoding, a deep neural network (Spike-Element-Wise-ResNet, SEW-ResNet) is further used to extract features from the diversity-encoded image, thereby further improving the accuracy. Based on the extracted features, the feature map of each object in the image is obtained, that is, Figure 4 The visual intermediate cognitive result in the figure V 11 、V 12 、V 13 etc. are the feature maps output by each channel.

[0103] See Figure 4 ,Furthermore, the visual cortex V3 model, the visual cortex V4 model, the visual cortex V5 model, the intraparietal sulcus model and the inferior temporal lobe cortex model are ,adopted to perform high-level cognition on the feature map and the ,question text to obtain the final cognitive result of each object in the ,image.

[0104] It should be noted that based on the collaborative mechanism of the ventral pathway and the dorsal pathway of the visual pathway, high-level cognition of the feature map and the question text is performed. The ventral pathway mainly includes the visual cortex V4 and the inferior temporal cortex (i.e., IT), and the dorsal pathway mainly includes the visual cortex V3, the visual cortex V5, and the intraparietal sulcus (i.e., IPS). Therefore, the visual cortex V3 model, the visual cortex V4 model, the visual cortex V5 model, the intraparietal sulcus model, and the inferior temporal cortex model can be constructed; among them, these models all use the LIF model.

[0105] See Figure 4 , the engineering process of the above model is:

[0106] 1) The visual cortex V4 model and the visual cortex V3 model are used to extract the object nodes and the relationships between the object nodes in the diversity-encoded image, and a scene graph is generated according to the relationships between the object nodes and the object nodes.

[0107] It should be noted that the visual cortex V3 model inputs masks and image features and outputs another mask, thus forming a relationship. Specifically, it receives the feature map extracted by the visual cortex V2 model and uses the attention module to locate prominent objects, that is, to assign greater weights to potential places in the map that can become object nodes, and then uses a series of extended convolution operations to expand the receptive field to the entire image, thereby providing the global context required for location information, thereby extracting the spatial features of the object.

[0108] The visual cortex V4 model builds a scene graph based on objects. The input is the feature map extracted by the visual cortex V2 model. Masks are assigned to allow the model to focus more on the area of ​​interest and ignore other irrelevant parts. This method can reduce noise and interference, thereby improving the accuracy of object recognition. Combined with the positional relationship extracted by the dorsal pathway, a scene graph is formed for each object. The scene graph contains object nodes and the relationship between object nodes (i.e., edge nodes). Object nodes are used to store the feature information of the object. Edge nodes are the coordinate differences between object nodes and are used to store positional relationships. These nodes will later embed the stored features according to the labels.

[0109] It should be noted that in the visual cortex V4 model, all possible combinations of object attributes are enumerated, and the attention allocation method is used to focus on the area of ​​the image containing objects with specified attributes. The image is taken as input and the heat map attention mask with a dimension of 1×H×W is output, which is completed by multiplying the input image features by the input mask elements, and then processing these focused features with several convolutions; where H represents height and W represents width.

[0110] Attention is that the mask intersection of red, cube, metal, and large objects constitutes a single object, and the scene graph is represented as a directed graph ( V , ε ),V Represents an object node, v i1 Representative i 1 object's features, ε represents the edge, ε i1j1 Representatives from i 1 object to j 1 object’s relational features. C Collect into a dictionary, using the embedding matrix D Map labels to d dimensional vectors and embed them to represent nodes and edges, and build a scene graph for each object based on the features extracted in the previous steps. b =softmax( Dq ) calculates the attention distribution, where q represents the problem, softmax is the softmax function, b The length or size length( b )= C and b c Indicates c The weights of the labels are then summed to capture the attention of nodes and edges:

[0111] ;

[0112] ;

[0113] In the formula, a i1 For the i 1 object node’s attention, W i1j1 Representatives from i 1 object to j 1 object edge attention, f 1 and g 1 is the summation function, vis is the query vector, representing a visual attribute (such as "color" is "green"), C i1 For the i 1 label of the object node, C i1j1 Representatives from i 1 object to j 1 label for the object edge.

[0114] 2) The visual cortex V5 model is used to parse the action of the question text, and the intraparietal sulcus model is used to plan the action parsing results. According to the planning results and the scene graph, a directed sequence of the scene graph is generated.

[0115] It should be noted that the visual cortex V5 model uses spatial relations and question features for action parsing, and then forms sequential processing in the intraparietal sulcus model. For datasets with real program annotations (such as CLEVR), we directly learn LSTM (Long Short-Term Memory Network) sequence-to-sequence models to convert word sequences into modular programs. However, most real-world datasets do not have layout annotations. In this case, a differentiable stack structure is used for soft module selection.

[0116] 3) According to the directed sequence of the scene graph, the inferior temporal cortex model is used to perform walk-back causation on the scene graph to obtain the final cognitive result of each object in the image.

[0117] It should be noted that the inferior temporal cortex model is able to form functionally specialized columns, perform cognitive functions in response to specific complex objects, and walk back the cause on the scene graph: β∈[0,1] N represents the attention weight vector of the object node, N is the dimension of the object node, and β i1 Indicates i The attention weight of an object node, the edge attention weight matrix is ​​И∈[0,1] N×N Indicates that i1j1 Representatives from i 1 object to j 1 object edge attention weight. The a priori module performs basic tasks and performs output by walking upstream in the scene graph.

[0118] The inferior temporal cortex model further forms a composite module in the complex cell layer according to the scene graph obtained from the image. The feature output module maps the intermediate attention to the hidden layer for feature representation, which is fed into the Softmax layer to predict the final answer, or fed into the module for further reasoning.

[0119] After obtaining the final cognitive result, the prefrontal cortex (PF) model is further used to infer the final cognitive result to obtain the answer corresponding to the image and question text.

[0120] It should be noted that see Figure 5 , Since the visual reasoning task is combinatorial, the problem is parsed into a sequence for module assembly, and finally the answer is predicted based on the visual cognition results. We believe that reasoning occurs in the area where the dorsal pathway and the ventral pathway converge, that is, the prefrontal layer. Because the information of the text modality can interact with the prefrontal model, the prefrontal model can also obtain the results of visual cognition. In this step, according to the progress of the directed sequence, combined with the visual walk causation, the output features of the module can be obtained, and these final features are input into the Softmax layer for answer prediction.

[0121] It should be noted that the human brain visual pathway model needs to be pre-trained before use, specifically using the cross entropy loss between the predicted answer and the true answer for training.

[0122] See Figure 5 , based on the initial predicted answer s , the answer is calibrated by measuring the similarity between the original image features and the reconstructed image features. ,question q is the word vector, and After connection, the global vector is embedded to obtain the embedded representation result c sq , and then c sq Pass it to GRU (Gate Recurrent Unit) to get the hidden state h sq :

[0123] ;

[0124] h sq =GRU( c sq );

[0125] Where Emb represents the global vector embedding.

[0126] From 0, 1 standard Gaussian distribution P ( z ) z ( z is a vector), y = f ( x ) means from vector x To vector y The linear weight matrix of h sq is the hidden vector of the answer and question content, and the potential distribution is a diagonal Gaussian distribution, z The mean and variance of the prior distribution of are parameterized as:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] In the formula, is the potential distribution, μ is the mean, σ is the standard deviation, f is the linear weight matrix, is a diagonal Gaussian distribution.

[0132] Will z Regenerate image features through multi-layer neural network V r :

[0133] h= RELU W 1 z + b 1);

[0134] V r =Sigmoid( W 2 h + b 2);

[0135] In the formula, W 1. W 2. b 1. b 2 are all trainable parameters, which are determined by Q ( z | h sq )and P ( z ) plus the KL divergence of the visual features (i.e., V 11 、V 12 、V 13 etc.) and reconstruction features V r The second reconstruction is to reconstruct the original image from the visual features and calculate the original image v and reconstruction features V r The L2 loss is used as the loss function for the second reconstruction.

[0136] The present invention aims at autonomous driving visual reasoning, constructs a cognitive process based on the human brain's visual pathway, uses a neural network to simulate the process of visual signals entering from the retina to the lateral geniculate body and then to the primary visual cortex, fully extracts the underlying features of the image, and lays the foundation for the subsequent semantic information cognition. Through the collaborative mechanism of the ventral pathway and the dorsal pathway of the visual pathway, the high-level semantic cognitive process of the image is opened to assist in understanding the logic of image composition. The present invention proposes a reasoning process based on the human brain's prefrontal lobe, adopts the LSTM sequence-to-sequence model, converts the word sequence of the question into a modular program, combines the image semantic cognitive process, forms a highly hierarchical reasoning process and predicts the answer.

[0137] The present invention realizes visual reasoning based on visual features and semantic cognition of visual pathways, fully opens up the neural cognitive process, and improves interpretability.

[0138] Figure 6 This is a schematic diagram of an embodiment of the autonomous driving visual reasoning and cognition device disclosed in the present invention. Figure 6 An embodiment of the present invention is a virtual device that can be loaded and executed by a controller of an autonomous driving system, including an acquisition module and a cognitive module.

[0139] The acquisition module of the embodiment is configured to acquire images in an autonomous driving scenario and question texts corresponding to the images.

[0140] The cognitive module of the embodiment is configured to obtain the answer corresponding to the image and the question text using a human brain visual pathway model based on the image and the question text corresponding to the image.

[0141] The working process of the human brain visual pathway model is as follows: using the retina model, lateral geniculate body model, visual cortex V1 model and visual cortex V2 model to perform low-level and intermediate cognition on the image to obtain a diversity-encoded image; using a pulse neural network to extract features from the diversity-encoded image, and based on the extracted features, obtain a feature map of each object in the image; using the visual cortex V3 model, visual cortex V4 model, visual cortex V5 model, intraparietal sulcus model and inferior temporal cortex model to perform high-level cognition on the feature map and question text to obtain the final cognitive result of each object in the image; using the prefrontal lobe model to infer the final cognitive result to obtain the answer corresponding to the image and the question text; wherein, each model in the human brain visual pathway model is used to imitate each cerebral cortex of the visual pathway, and a neuron model is used to model the cells of the cerebral cortex.

[0142] It should be noted that in the human brain visual pathway model of the cognitive module, the retinal model, lateral geniculate body model, visual cortex V1 model and visual cortex V2 model are used to perform low-level and intermediate cognition of images to obtain diversely encoded images, including: using the retinal model to convert images into pulse signals; using the lateral geniculate body model to integrate the pulse signals; using the visual cortex V1 model to perform multi-channel feature extraction on the output signal of the lateral geniculate body model, and encoding the features of the outputs of different pathways; using the visual cortex V2 model to further extract the encoded features to obtain diversely encoded images.

[0143] It should be noted that, in the human brain visual pathway model of the cognitive module, the visual cortex V3 model, the visual cortex V4 model, the visual cortex V5 model, the intraparietal sulcus model and the inferior temporal cortex model are used to perform high-level cognition on the feature map and the question text to obtain the final cognitive result of each object in the image, including: using the visual cortex V4 model and the visual cortex V3 model to respectively extract the object nodes and the relationship between the object nodes in the image after diversity encoding, and generating a scene graph according to the relationship between the object nodes; using the visual cortex V5 model to perform action analysis on the question text, using the intraparietal sulcus model to plan the action analysis results, and generating a directed sequence of the scene graph according to the planning results and the scene graph; using the inferior temporal cortex model to perform walk-back causation on the scene graph according to the directed sequence of the scene graph to obtain the final cognitive result of each object in the image.

[0144] The above-mentioned device uses a human brain visual pathway model that imitates the visual pathway to perform visual reasoning and cognition in autonomous driving, and uses a neuron model to simulate the process of visual signals entering from the retina to the lateral geniculate body and then to the primary visual cortex, which is conducive to extracting the underlying features of the image and laying the foundation for subsequent semantic information cognition. The ventral pathway and dorsal pathway of the visual pathway are simulated by a neuron model, which is conducive to high-level cognition of feature maps and question texts and assists in understanding the logic of image composition. The prefrontal lobe of the human brain is simulated by a neuron model, which is conducive to inferring the answers corresponding to the image and question text, realizing visual reasoning based on the visual features and semantic cognition of the visual pathway, fully opening up the neural cognitive process and improving interpretability.

[0145] Based on the same technical solution, the present disclosure also relates to a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes an autonomous driving visual reasoning cognition method.

[0146] Based on the same technical solution, the present disclosure also relates to a computer device, including one or more processors and one or more memories, one or more programs are stored in the one or more memories and are configured to be executed by one or more processors, and the one or more programs include instructions for executing an autonomous driving visual reasoning cognition method.

[0147] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A visual reasoning and cognition method for autonomous driving, characterized in that: include: Obtain images in autonomous driving scenarios and corresponding question texts; According to the image and the corresponding question text, the human brain visual pathway model is used to obtain the answer corresponding to the image and the question text; Among them, the working process of the human brain visual pathway model is: The image is converted into a pulse signal by using a retina model, the pulse signal is integrated by using a lateral geniculate body model, the output signal of the lateral geniculate body model is extracted by using a visual cortex V1 model, and the features of the output of different pathways are encoded, and the encoded features are further extracted by using a visual cortex V2 model to obtain a diversified encoded image; wherein, in the visual cortex V1 model, when receiving the output of the lateral geniculate body model, a feedback pulse is sent to the lateral geniculate body model to adjust the firing rate of the output layer of the lateral geniculate body model; Perform feature extraction on the diversity-encoded image, and obtain a feature map of each object in the image based on the extracted features; The visual cortex V4 model and the visual cortex V3 model are used to extract the object nodes and the relationship between the object nodes in the diversity-encoded image, respectively. According to the relationship between the object nodes and the object nodes, a scene graph is generated. The visual cortex V5 model is used to perform action parsing on the question text. The intraparietal sulcus model is used to plan the action parsing results. According to the planning results and the scene graph, a directed sequence of the scene graph is generated. According to the directed sequence of the scene graph, the inferior temporal cortex model is used to perform walk-back causation on the scene graph to obtain the final cognitive result of each object in the image. The prefrontal lobe model is used to infer the final cognitive results and obtain the answers corresponding to the image and question text; Each model in the human brain visual pathway model is used to imitate each cerebral cortex of the visual pathway, and a neuron model is used to model cells of the cerebral cortex.

2. The autonomous driving visual reasoning cognition method according to claim 1, characterized in that: Models of the human visual pathway also include a priori alignment networks; Before performing low-level and mid-level cognition on the image, a prior alignment network is used to align the image and question text a priori.

3. An autonomous driving visual reasoning and cognition device, characterized in that: include: An acquisition module is used to obtain images in autonomous driving scenarios and corresponding question texts. The cognitive module uses the human brain visual pathway model to obtain the answer corresponding to the image and the question text based on the image and the question text; Among them, the working process of the human brain visual pathway model is: The image is converted into a pulse signal by using a retina model, the pulse signal is integrated by using a lateral geniculate body model, the output signal of the lateral geniculate body model is extracted by using a visual cortex V1 model, and the features of the output of different pathways are encoded, and the encoded features are further extracted by using a visual cortex V2 model to obtain a diversified encoded image; wherein, in the visual cortex V1 model, when receiving the output of the lateral geniculate body model, a feedback pulse is sent to the lateral geniculate body model to adjust the firing rate of the output layer of the lateral geniculate body model; Perform feature extraction on the diversity-encoded image, and obtain a feature map of each object in the image based on the extracted features; The visual cortex V4 model and the visual cortex V3 model are used to extract the object nodes and the relationship between the object nodes in the diversity-encoded image, respectively. According to the relationship between the object nodes and the object nodes, a scene graph is generated. The visual cortex V5 model is used to perform action parsing on the question text. The intraparietal sulcus model is used to plan the action parsing results. According to the planning results and the scene graph, a directed sequence of the scene graph is generated. According to the directed sequence of the scene graph, the inferior temporal cortex model is used to perform walk-back causation on the scene graph to obtain the final cognitive result of each object in the image. The prefrontal lobe model is used to infer the final cognitive results and obtain the answers corresponding to the image and question text; Each model in the human brain visual pathway model is used to imitate each cerebral cortex of the visual pathway, and a neuron model is used to model cells of the cerebral cortex.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes any one of the methods of claims 1 to 2.

5. A computer device, characterized in that: include: One or more processors and one or more memories, one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods described in claims 1 to 2.

Images