Autonomous driving perception method, device and storage medium based on implicit graph reasoning

Abstract

The present invention discloses an autonomous driving perception method, device and storage medium based on implicit graph reasoning, including: obtaining a number of multimodal information related to autonomous driving; according to each single-modal information, using a trained implicit graph inference model to infer the corresponding implicit graph structure; based on the implicit graph structure, modeling is performed through a number of prior knowledge to achieve autonomous driving perception. The present invention extracts the implicit graph structure of each modal information from the multimodal information of complex high-dimensional data, and the structured semantic expression of the implicit graph structure obtained by reasoning is often more in line with human cognition, which can improve the interpretability of autonomous driving perception. Based on the implicit graph structure, fine modeling is performed through a number of prior knowledge to achieve a more refined and interpretable representation than a vector, thereby improving the perception ability of the autonomous driving system for complex environments.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving perception technology, and specifically relates to an autonomous driving perception method, device and storage medium based on implicit graph reasoning. Background Art

[0002] Autonomous driving is a major trend in the future transportation field. It can not only improve driving safety and reduce the incidence of traffic accidents, but also improve traffic congestion, improve traffic efficiency, save energy and reduce emissions. In the development of autonomous driving technology, autonomous driving perception technology plays a very important role in the entire autonomous driving technology pipeline. Its main function is to perceive and understand the environment around the vehicle in real time and pass this information to other modules of the autonomous driving system, such as decision-making and planning modules. Perception environment technology obtains information about the surrounding environment through various sensors and devices, converts it into a computer-readable form, and uses various algorithms to process and analyze this information, thereby realizing the perception, understanding and recognition of the surrounding environment.

[0003] Existing autonomous driving perception methods ignore the potential relationships within data samples when extracting features from acquired unimodal information, and are unable to accurately and efficiently estimate the structural information within the samples. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an autonomous driving perception method, device and storage medium based on implicit graph reasoning.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention discloses an autonomous driving perception method based on implicit graph reasoning, comprising:

[0007] Step S1: obtaining some multimodal information related to autonomous driving;

[0008] Step S2: Based on each unimodal information, the trained implicit graph inference model is used to infer the corresponding implicit graph structure;

[0009] Step S3: Based on the implicit graph structure, modeling is performed through some prior knowledge to achieve autonomous driving perception.

[0010] Based on the above technical solution, the following improvements can be made:

[0011] As a preferred solution, the training of the implicit graph inference model includes:

[0012] Step A: Get several training samples X;

[0013] Step B: Establish an implicit graph inference model, treat the implicit graph structure as a hidden variable, and the prior probability distribution of the implicit graph structure is p(A)) = Π uv p(A uv );

[0014] Where: A is the adjacency matrix of the graph structure;

[0015] A uv Indicates the connection relationship between point u and point v in the graph. When the value is 0, it means there is no edge, and when the value is 1, it means there is an edge.

[0016] Step C: Based on the prior probability distribution, use the implicit graph inference model to predict the label Y and generate the data distribution p θ (Y|X,A);

[0017] Where: θ represents the parameters of the implicit graph inference model.

[0018] As a preferred solution, the parameter θ of the implicit graph inference model is optimized by:

[0019] Directly assume a learnable free parameter {α uv};

[0020] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0021] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0022] q φ (A) = ∏ u,v q φ (A uv ),where q φ (A uv )=Bern(A uv |α uv ),α uv >0;

[0023] The expectation maximization algorithm is used for learning;

[0024] For step E, the model parameters θ are fixed and the parameters φ of the implicit graph structure are updated as follows;

[0025]

[0026] Where: KL represents Kullback-Leibler divergence;

[0027] For the M-step, the implicit graph structure is fixed and the parameters θ of the model are updated as follows;

[0028]

[0029] As a preferred solution, the parameter θ of the implicit graph inference model is optimized by:

[0030] Each potential edge is indirectly parameterized using the representation vector of each node in the implicit graph;

[0031] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0032] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0033] q φ (A uv |x iu ,x iv )=Bern(A uv |β uv ),β uv =σ([(W1x iu ) T ,(W2x iv ) T ]w);

[0034] Where: σ represents the sigmoid function, which is used to constrain the output value between 0 and 1;

[0035] x iu Represents the edge connecting point i and point u in the graph;

[0036] x iu Represents the edge connecting point i and point u in the graph;

[0037] β uv Indicates the probability value that there is an edge between point u and point v in the graph;

[0038] W1 and W2 are matrix parameters;

[0039] Using the variational inference algorithm, the parameters φ of the implicit graph structure and the parameters θ of the model are jointly updated as follows;

[0040]

[0041] Where: KL represents Kullback-Leibler divergence.

[0042] In a second aspect, the present invention further discloses an autonomous driving perception device based on implicit graph reasoning, comprising:

[0043] An acquisition module, used to acquire a number of multimodal information related to autonomous driving;

[0044] The reasoning module is used to obtain the corresponding implicit graph structure based on each single-modal information using the trained implicit graph inference model;

[0045] The perception module is used to achieve autonomous driving perception by modeling based on implicit graph structure and some prior knowledge.

[0046] As a preferred solution, the autonomous driving perception device further includes: a training module for training an implicit graph inference model;

[0047] The training modules include:

[0048] An acquisition unit, used for acquiring a number of training samples X;

[0049] The model building unit is used to build an implicit graph inference model, treating the implicit graph structure as a hidden variable. The prior probability distribution of the implicit graph structure is p(A)=П uv p(A uv );

[0050] Where: A is the adjacency matrix of the graph structure;

[0051] A uv Indicates the connection relationship between point u and point v in the graph. When the value is 0, it means there is no edge, and when the value is 1, it means there is an edge.

[0052] The prediction unit is used to predict the label Y based on the prior probability distribution and generate the data distribution p using the implicit graph inference model. θ (Y|X,A);

[0053] Where: θ represents the parameters of the implicit graph inference model.

[0054] As a preferred solution, the training module uses the following method to optimize the parameter θ of the implicit graph inference model;

[0055] Directly assume a learnable free parameter {α uv};

[0056] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0057] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0058] q φ (A) = ∏ u,v q φ (A uv),where q φ (A uv )=Bern(A uv |α uv ),α uv >0;

[0059] The expectation maximization algorithm is used for learning;

[0060] For step E, the model parameters θ are fixed and the parameters φ of the implicit graph structure are updated as follows;

[0061]

[0062] Where: KL represents Kullback-Leibler divergence;

[0063] For the M-step, the implicit graph structure is fixed and the parameters θ of the model are updated as follows;

[0064]

[0065] As a preferred solution, the training module uses the following method to optimize the parameter θ of the implicit graph inference model;

[0066] Each potential edge is indirectly parameterized using the representation vector of each node in the implicit graph;

[0067] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0068] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0069] q φ (A uv |x iu ,x iv )=Bern(A uv |β uv ),β uv =σ([(W1x iu ) T ,(W2x iv ) T ]w);

[0070] Where: σ represents the sigmoid function, which is used to constrain the output value between 0 and 1;

[0071] x iu Represents the edge connecting point i and point u in the graph;

[0072] x iu Represents the edge connecting point i and point u in the graph;

[0073] β uv Indicates the probability value that there is an edge between point u and point v in the graph;

[0074] W1 and W2 are matrix parameters;

[0075] Using the variational inference algorithm, the parameters φ of the implicit graph structure and the parameters θ of the model are jointly updated as follows;

[0076]

[0077] Where: KL represents Kullback-Leibler divergence.

[0078] In a third aspect, the present invention further discloses a storage medium, which includes a computer-readable computer program. When the computer program is executed by one or more processors, it implements any of the above-mentioned autonomous driving perception methods based on implicit graph reasoning.

[0079] The present invention discloses an automatic driving perception method, device and storage medium based on implicit graph reasoning, which has the following beneficial effects:

[0080] The present invention extracts the implicit graph structure of each modal information from the multimodal information of complex high-dimensional data. The structured semantic expression of the implicit graph structure obtained by reasoning is often more consistent with human cognition, which can improve the interpretability of autonomous driving perception. Based on the implicit graph structure, fine modeling is performed through a number of prior knowledge to achieve a more refined and interpretable representation than a vector, thereby improving the perception ability of the autonomous driving system for complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0082] Figure 1 A flowchart of an autonomous driving perception method provided in an embodiment of the present invention.

[0083] Figure 2 A schematic diagram of implicit graph structure learning and reasoning based on the EM algorithm and the ELBO algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0087] In order to achieve the purpose of the present invention, some embodiments of the autonomous driving perception method, device and storage medium based on implicit graph reasoning are as follows: Figure 1 As shown, the autonomous driving perception method includes:

[0088] Step S1: obtaining a number of multimodal information related to autonomous driving, such as point cloud, video, map, etc., from millimeter wave radar sensors, lidar sensors, ultrasonic radar sensors, cameras, and other devices;

[0089] Step S2: Based on each unimodal information, the trained implicit graph inference model is used to infer the corresponding implicit graph structure;

[0090] Step S3: Based on the implicit graph structure, detailed modeling is performed through prior knowledge such as macroscopic distribution (such as symmetry) and local geometric properties to achieve autonomous driving perception.

[0091] The training of the implicit graph inference model mentioned above includes:

[0092] Step A: Get several training samples X;

[0093] Step B: Establish an implicit graph inference model, treat the implicit graph structure as a hidden variable, and the prior probability distribution of the implicit graph structure is p(A)=Π uv p(A uv );

[0094] Where: A is the adjacency matrix of the graph structure;

[0095] A uv Indicates the connection relationship between point u and point v in the graph. When the value is 0, it means there is no edge, and when the value is 1, it means there is an edge.

[0096] Step C: Based on the prior probability distribution, use the implicit graph inference model to predict the label Y and generate the data distribution p θ (Y|X,A);

[0097] Where: θ represents the parameters of the implicit graph inference model.

[0098] For each modal information, a graph structure composed of internal input features is considered for training sample X, so as to utilize the correlation between the internal features of the sample to enhance the expressiveness and interpretability of the model, thereby improving the performance and expression of autonomous driving perception. Specifically, in order to extract implicit graph structures from high-dimensional data, the Bayesian learning method is used to model the distribution of implicit graph structures.

[0099] First, the implicit graph structure is regarded as a latent variable, and the prior probability distribution of the graph structure is assumed to be p(A), which describes the approximate description of the implicit graph structure based on the prior in specific scenarios and tasks.

[0100] In some specific embodiments, common prior knowledge includes: sparsity, low rank, local connectivity, etc. of the graph structure.

[0101] In addition, the data distribution generated by the implicit graph inference model for predicting the label Y can be expressed as p θ (Y|X,A). We hope to obtain the posterior distribution corresponding to the estimated implicit graph structure as p(A|X,Y)∝p(A)p θ (Y|X,A), which represents the distribution of the ideal implicit graph structure.

[0102] It is very difficult to directly calculate the posterior distribution p(A|X,Y) (usually it is necessary to integrate high-dimensional continuous variables), so here we consider introducing a variational distribution q φ (A) to learn the implicit structure.

[0103] like Figure 2 As shown, two methods for optimizing the parameters θ of the implicit graph inference model are introduced below, specifically method 1 and method 2.

[0104] Method 1

[0105] Directly assume a learnable free parameter {α uv};

[0106] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0107] The Bernoulli distribution is used to model the graph structure probability, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0108] q φ (A)=Π u,v q φ (A uv ),where q φ (A uv)=Bern(A uv |α uv ),α uv >0;

[0109] Use the Expectation Maximization (EM) algorithm for learning;

[0110] For step E, the model parameters θ are fixed and the parameters φ of the implicit graph structure are updated as follows;

[0111]

[0112] Where: KL represents Kullback-Leibler divergence;

[0113] For the M-step, the implicit graph structure is fixed and the parameters θ of the model are updated as follows;

[0114]

[0115] The advantage of method 1 is that the model has enough capacity to learn any graph structure. However, the cost is that it is easy to cause the model to be over-parameterized and the training is time-consuming. Therefore, method 1 is suitable for scenarios with small graphs and large number of data sets.

[0116] The above θ (Y|A,X) and p θ (Y|X,A) have the same meaning.

[0117] Method 2

[0118] Each potential edge is indirectly parameterized using the representation vector of each node in the implicit graph;

[0119] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0120] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0121] q φ (A uv |x iu ,x iv )=Bern(A uv |β uv ),β uv =σ([(W1x iu ) T ,(W2x iv ) T ]w);

[0122] Where: σ represents the sigmoid function, which is used to constrain the output value between 0 and 1;

[0123] x iu Represents the edge connecting point i and point u in the graph;

[0124] x iu Represents the edge connecting point i and point u in the graph;

[0125] β uv Indicates the probability value that there is an edge between point u and point v in the graph;

[0126] W1 and W2 are matrix parameters;

[0127] Using the variational inference (Evidence Lower Bound, ELBO) algorithm, the parameters φ of the implicit graph structure and the parameters θ of the model are jointly updated as follows;

[0128]

[0129] Where: KL represents Kullback-Leibler divergence.

[0130] The first term of the objective function above is It can be estimated by the loss function of supervised learning (such as cross entropy loss for classification tasks, or squared error loss for regression tasks). The second term KL(q φ (A)|p(A)) can be understood as a regular term.

[0131] The advantage of the above method 2 is that it reduces the number of model parameters, improves the training speed, and makes the model more scalable. However, this method sacrifices the expressiveness of the model to a certain extent. Therefore, method 2 is more suitable for scenarios with larger graphs and fewer data set samples.

[0132] The first method assumes a free parameter for each edge, and the second method uses node representation to estimate the existence of edges. The above schemes flexibly incorporate prior information, improving the interpretability and even controllability of the modeling and learning process.

[0133] In summary, in addition to processing data with explicit graph structures (such as social networks and maps), the present invention uses a data-driven approach combined with priors and rules to perform implicit graph structure inference on general unstructured data including point clouds and videos, and has the ability to extract features and represent abstractions for a variety of input samples in different scenarios and tasks.

[0134] The present invention uses graph structures to represent complex high-dimensional data, and its structured semantic expression is often more in line with human cognition. The potential graph structure behind the data can be finely modeled through macroscopic distribution (such as symmetry) and local geometric properties, which can achieve a more refined and interpretable representation than vectors. It can provide a reference for explaining the working principle of the model and the implicit laws behind the data, enhance the interpretability of the model, and reduce the risk of use.

[0135] It is worth noting that an embodiment of the present invention further discloses an autonomous driving perception device based on implicit graph reasoning, including:

[0136] An acquisition module, used to acquire a number of multimodal information related to autonomous driving;

[0137] The reasoning module is used to obtain the corresponding implicit graph structure based on each single-modal information using the trained implicit graph inference model;

[0138] The perception module is used to achieve autonomous driving perception by modeling based on implicit graph structure and some prior knowledge.

[0139] The autonomous driving perception device further includes: a training module for training an implicit graph inference model;

[0140] The training modules include:

[0141] An acquisition unit, used for acquiring a number of training samples X;

[0142] The model building unit is used to build an implicit graph inference model, treating the implicit graph structure as a hidden variable. The prior probability distribution of the implicit graph structure is p(A)=Π uv p(A uv );

[0143] Where: A is the adjacency matrix of the graph structure;

[0144] A uv Indicates the connection relationship between point u and point v in the graph. When the value is 0, it means there is no edge, and when the value is 1, it means there is an edge.

[0145] The prediction unit is used to predict the label Y based on the prior probability distribution and generate the data distribution p using the implicit graph inference model. θ (Y|X,A);

[0146] Where: θ represents the parameters of the implicit graph inference model.

[0147] The above training module uses the following method to optimize the parameters θ of the implicit graph inference model;

[0148] Directly assume a learnable free parameter {α uv};

[0149] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0150] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0151] q φ (A)=Π u,v q φ (A uv ),where q φ (A uv )=Bern(A uv |α uv ),α uv >0;

[0152] The expectation maximization algorithm is used for learning;

[0153] For step E, the model parameters θ are fixed and the parameters φ of the implicit graph structure are updated as follows;

[0154]

[0155] Where: KL represents Kullback-Leibler divergence;

[0156] For the M-step, the implicit graph structure is fixed and the parameters θ of the model are updated as follows;

[0157]

[0158] In other embodiments, the training module optimizes the parameter θ of the implicit graph inference model using the following method;

[0159] Each potential edge is indirectly parameterized using the representation vector of each node in the implicit graph;

[0160] Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure;

[0161] The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge:

[0162] q φ (A uv |x iu ,x iv )=Bern(A uv |β uv ),β uv =σ([(W1x iu )T ,(W2x iv ) T ]w);

[0163] Where: σ represents the sigmoid function, which is used to constrain the output value between 0 and 1;

[0164] x iu Represents the edge connecting point i and point u in the graph;

[0165] x iu Represents the edge connecting point i and point u in the graph;

[0166] β uv Indicates the probability value that there is an edge between point u and point v in the graph;

[0167] W1 and W2 are matrix parameters;

[0168] Using the variational inference algorithm, the parameters φ of the implicit graph structure and the parameters θ of the model are jointly updated as follows;

[0169]

[0170] Where: KL represents Kullback-Leibler divergence.

[0171] In this embodiment, the specific content of the autonomous driving perception device is similar to the content of the autonomous driving perception method disclosed in the above embodiment, and will not be repeated here.

[0172] In some other embodiments, the present invention further discloses a storage medium, which includes a computer-readable computer program. When the computer program is executed by one or more processors, it implements the autonomous driving perception method based on implicit graph reasoning disclosed in any of the above embodiments.

[0173] The present invention discloses an automatic driving perception method, device and storage medium based on implicit graph reasoning, which has the following beneficial effects:

[0174] The present invention extracts the implicit graph structure of each modal information from the multimodal information of complex high-dimensional data. The structured semantic expression of the implicit graph structure obtained by reasoning is often more consistent with human cognition, which can improve the interpretability of autonomous driving perception. Based on the implicit graph structure, fine modeling is performed through macroscopic distribution and local geometric properties to achieve a more refined and interpretable representation than a vector, thereby improving the perception ability of the autonomous driving system for complex environments.

[0175] The present invention can infer implicit graph structures from general unstructured data such as point clouds and videos. The present invention can be applied to real scenarios of autonomous driving, including but not limited to campus scenarios, vehicle-mounted fields, traffic radar fields, and other scenarios.

[0176] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0177] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An autonomous driving perception method based on implicit graph reasoning, characterized in that: include: Step S1: Acquire some multimodal information related to autonomous driving, wherein the multimodal information includes: point cloud information, video information, and map information; Step S2: Based on each unimodal information, the trained implicit graph inference model is used to infer the corresponding implicit graph structure; Step S3: Based on the implicit graph structure, modeling is performed through some prior knowledge to achieve autonomous driving perception; The training of the implicit graph inference model includes: Step A: Get several training samples X; Step B: Establish an implicit graph inference model, treat the implicit graph structure as a hidden variable, and the prior probability distribution of the implicit graph structure is p(A) = ∏ uv p(A uv ); Where: A is the adjacency matrix of the graph structure; A uv Indicates the connection relationship between point u and point v in the graph. When the value is 0, it means there is no edge, and when the value is 1, it means there is an edge. Step C: Based on the prior probability distribution, use the implicit graph inference model to predict the label Y and generate the data distribution p θ (Y|X,A); Where: θ represents the parameters of the implicit graph inference model; The parameters θ of the implicit graph inference model are optimized by: Directly assume a learnable free parameter {α uv }; Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure; The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge: q φ (A)=∏ u,v q φ (A uv ),where q φ (A uv )=Bern(A uv |α uv ),α uv >0; The expectation maximization algorithm is used for learning; For step E, the model parameters θ are fixed and the parameters φ of the implicit graph structure are updated as follows; Where: KL represents Kullback-Leibler divergence; For the M-step, the implicit graph structure is fixed and the parameters θ of the model are updated as follows; or, The parameters θ of the implicit graph inference model are optimized by: Each potential edge is indirectly parameterized using the representation vector of each node in the implicit graph; Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure; The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge: Where: σ represents the sigmoid function, which is used to constrain the output value between 0 and 1; x iu Represents the edge connecting point i and point u in the graph; x iu Represents the edge connecting point i and point u in the graph; β uv Indicates the probability value that there is an edge between point u and point v in the graph; W1 and W2 are matrix parameters; Using the variational inference algorithm, the parameters φ of the implicit graph structure and the parameters θ of the model are jointly updated as follows; Where: KL represents Kullback-Leibler divergence.

2. An autonomous driving perception device based on implicit graph reasoning, characterized in that: include: An acquisition module, used to acquire a number of multimodal information related to autonomous driving, wherein the multimodal information includes: point cloud information, video information, and map information; The reasoning module is used to obtain the corresponding implicit graph structure based on each single-modal information using the trained implicit graph inference model; A perception module is used to achieve autonomous driving perception by modeling based on an implicit graph structure through a number of prior knowledge; the autonomous driving perception device also includes: a training module, which is used to train the implicit graph inference model; The training module includes: An acquisition unit, used for acquiring a number of training samples X; The model building unit is used to build an implicit graph inference model, treating the implicit graph structure as a hidden variable. The prior probability distribution of the implicit graph structure is p(A) = ∏ uv p(A uv ); Where: A is the adjacency matrix of the graph structure; A uv Indicates the connection relationship between point u and point v in the graph. When the value is 0, it means there is no edge, and when the value is 1, it means there is an edge. The prediction unit is used to predict the label Y based on the prior probability distribution and generate the data distribution p using the implicit graph inference model. θ (Y|X,A); Where: θ represents the parameters of the implicit graph inference model; The training module optimizes the parameter θ of the implicit graph inference model using the following method; Directly assume a learnable free parameter {α uv }; Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure; The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge: q φ (A)=∏ u,v q φ (A uv ),where q φ (A uv )=Bern(A uv |α uv ),α uv >0; The expectation maximization algorithm is used for learning; For step E, the model parameters θ are fixed and the parameters φ of the implicit graph structure are updated as follows; Where: KL represents Kullback-Leibler divergence; For the M-step, the implicit graph structure is fixed and the parameters θ of the model are updated as follows; or, The training module optimizes the parameter θ of the implicit graph inference model using the following method; Each potential edge is indirectly parameterized using the representation vector of each node in the implicit graph; Introducing the variational distribution q φ (A) is used to learn the implicit structure, where φ represents the parameters of the implicit graph structure; The Bernoulli distribution is used to model the probability of the graph structure, and the probability of the overall graph structure is decomposed into the probability of each edge: q φ (A uv |x iu ,x iv )=Bern(A uv |b uv ),b uv =σ([(W1x iu ) T ,(W2x iv ) T ]w); Where: σ represents the sigmoid function, which is used to constrain the output value between 0 and 1; x iu Represents the edge connecting point i and point u in the graph; x iu Represents the edge connecting point i and point u in the graph; β uv Indicates the probability value that there is an edge between point u and point v in the graph; W1 and W2 are matrix parameters; Using the variational inference algorithm, the parameters φ of the implicit graph structure and the parameters θ of the model are jointly updated as follows; Where: KL represents Kullback-Leibler divergence.

3. A storage medium, characterized in that The storage medium includes a computer-readable computer program, which, when executed by one or more processors, implements the autonomous driving perception method based on implicit graph reasoning described in claim 1.

Images