Pure vision automatic driving environment sensing method and system based on variable depth mechanism

By employing a pure vision-based autonomous driving environment perception method based on a variable depth mechanism, multi-view image processing and temporal difference are used to generate a view frustum feature matrix and map it to the BEV space. This solves the problem of inaccurate object depth estimation and achieves efficient environment perception and improved navigation accuracy.

CN119048998APending Publication Date: 2024-11-29HEFEI INSTITUTE OF PHYSICAL SCIENCE CHINESE ACADEMY OF SCIENCES
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410973207.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing image-based autonomous driving environmental perception methods suffer from inaccurate object depth estimation, while radar-based methods are costly and image-radar fusion methods are complex and difficult to deploy on resource-constrained devices.

Method used

A pure vision-based autonomous driving environment perception method based on a variable depth mechanism is adopted. By acquiring original images from different perspectives, processing them to generate multi-view features, performing temporal difference and depth information extraction, generating a view frustum feature matrix, and mapping it to the BEV space, the result is simplified into an environment perception result.

Benefits of technology

It improves the accuracy of object depth estimation, simplifies computational resource requirements, provides more accurate environmental perception results, and enhances the safety and navigation accuracy of autonomous driving systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048998A_ABST
    Figure CN119048998A_ABST
Patent Text Reader

Abstract

The invention discloses a pure vision automatic driving environment perception method and system based on a variable depth mechanism, and the method comprises the steps: obtaining original images of different visual angles around a driving subject, and carrying out the processing of the original images of different visual angles, and obtaining a multi-visual-angle feature; performing difference processing on the multi-view feature at the current moment and the multi-view feature at the previous moment to obtain a time sequence difference feature containing time sequence information; performing depth information extraction on the time sequence difference features by using a variable depth mechanism to obtain variable depth distribution; performing outer product operation on the variable depth distribution in the time sequence difference features to generate a view cone feature matrix; mapping the view cone feature matrix to a BEV space with the vehicle as the center, and generating an environment perception result; according to the invention, the accuracy of object depth estimation in image-based automatic driving environment perception is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a pure visual autonomous driving environment perception method and system based on a variable depth mechanism. Background Art

[0002] With the advancement of technology, self-driving cars have gradually become the backbone of modern transportation systems. This technology uses sensors and computing systems to achieve autonomous driving of vehicles, aiming to alleviate traffic congestion and improve traffic safety. In the field of autonomous driving, the accuracy of environmental perception will directly affect the quality of subsequent decision-making, so accurate environmental perception is crucial.

[0003] Most of the vehicles with autonomous driving functions on the market now use radar-based perception methods and perception methods based on image and radar data fusion. However, the above perception methods all have inconveniences: although the radar-based perception method can provide accurate depth information, this method relies on the performance of radar hardware and has high requirements on the technical specifications and quality of the sensor. Radar sensors, especially high-precision laser radars, are usually very expensive, which will greatly increase the overall cost of the autonomous driving system; the method based on image and radar point cloud fusion requires two different types of data (image and point cloud) to be accurately synchronized to ensure the accuracy of data fusion. For example, the driving environment perception method combining image recognition and laser radar point cloud segmentation proposed in the patent application document with publication number CN113269040A is to achieve environmental perception by using the fusion of image and radar point cloud, but this method greatly increases the complexity of the system, and any sensor failure may seriously affect the performance of the entire perception system.

[0004] In contrast, image-based deep learning environmental perception methods are economical and can capture rich visual information. They are a differentiated highlight in the current competition in the field of autonomous driving, and are also an effective technical means to improve the efficiency and reliability of autonomous driving environmental perception. In a pure visual perception system, there is a proportional relationship between the size of objects in the image and their actual size, and this relationship is significantly affected by the distance between the object and the camera. However, since the image itself lacks depth information, it becomes difficult to judge the depth of the object directly from the image, which in turn affects the accuracy of object perception. This is a major challenge faced by image-based autonomous driving environmental perception methods.

[0005] In the related art, a patent application document with publication number CN118172618A proposes an end-to-end autonomous driving planning method. First, the purpose of this solution is to plan the autonomous driving vehicle through multi-view images instead of directly outputting the perceived results. Secondly, this solution adopts a top-down information processing strategy, that is, from global to local. First, a global, cross-view 3D spatial representation is constructed, and then the 3D representation is used to query the information of each individual image. This top-down strategy usually relies on complex models and algorithms, such as using the global perception ability of the Transformer model to mine the connection between different position information through the attention mechanism. When using the Transformer model, a large amount of computing resources are usually required due to the need to perform global attention mechanism operations, which makes it difficult to deploy on resource-constrained devices. In addition, due to the need to process a large amount of global information, more training data and tuning are required to optimize performance when using the Transformer model. In addition, after acquiring the multi-view image features, this scheme aligns the historical information stored in the memory queue with the coordinate system of the current time. Taking the A in "ABC" and "BAC" as an example, alignment means that the first A and the second A are placed in the same position through processing instead of performing a difference.

[0006] Therefore, there is an urgent need to improve the accuracy of object depth estimation in image-based autonomous driving environment perception through simpler deep learning methods. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to improve the accuracy of object depth estimation in image-based autonomous driving environment perception.

[0008] The present invention solves the above technical problems by the following technical means:

[0009] The present invention proposes a pure visual autonomous driving environment perception method based on a variable depth mechanism, the method comprising:

[0010] Acquire original images of different perspectives around the driving subject, and process the original images of different perspectives to obtain multi-perspective features;

[0011] Performing differential processing on the multi-view features at the current moment and the multi-view features at the previous moment to obtain a time series differential feature containing time series information;

[0012] Using a variable depth mechanism to extract depth information from the temporal difference feature to obtain a variable depth distribution;

[0013] Performing an outer product operation on the variable depth distribution and the temporal difference feature to generate a frustum feature matrix;

[0014] The cone feature matrix is ​​mapped to the BEV space centered on the ego vehicle to generate an environmental perception result.

[0015] Furthermore, the processing of original images of different viewing angles to obtain multi-view features includes:

[0016] The backbone network is used to process the original images from different perspectives to obtain multi-scale features;

[0017] The multi-scale features are fused to obtain the multi-view features.

[0018] Furthermore, the multi-view features at the current moment are differentially processed with the multi-view features at the previous moment to obtain a time series differential feature containing time series information, which is expressed as follows:

[0019]

[0020] In the formula, and They represent the multi-view features at time t and time t-1, N represents the number of viewpoints, Conv represents the convolution operation, T t Represents the temporal difference feature.

[0021] Furthermore, the step of extracting depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution includes:

[0022] Using the first linear layer to process the temporal difference features, identifying M key depths where the object is most likely to exist;

[0023] Using a second linear layer to learn the probability distribution of objects at the M key depths;

[0024] The M key depths are respectively fused with the corresponding probability distributions to obtain a variable depth distribution, where the variable depth distribution is a weighted distance distribution result.

[0025] Furthermore, the variable depth is distributed on the temporal difference feature to perform an outer product operation to generate a frustum feature matrix, which is expressed as follows:

[0026] c d =c×d t

[0027] In the formula, c d is the cone feature matrix, c represents the feature vector of the temporal difference feature, d t represents the transpose of the depth vector d of the variable depth profile.

[0028] Furthermore, mapping the visual cone feature matrix to the BEV space centered on the vehicle to generate an environmental perception result includes:

[0029] Mapping the feature points in the cone feature matrix to the BEV space through the camera's intrinsic parameter matrix K and extrinsic parameter matrix R;

[0030] The BEV space is divided into grids, each grid corresponds to a cylinder, and the feature points after spatial transformation are assigned to the nearest cylinder according to their positions in the BEV space;

[0031] The feature points in each column are averaged and pooled to generate the environment perception result.

[0032] Furthermore, original images of different viewing angles are acquired through cameras of different viewing angles equipped by the driving subject.

[0033] In addition, the present invention also proposes a pure visual autonomous driving environment perception system based on a variable depth mechanism, the system comprising:

[0034] An image acquisition module is used to acquire original images of different perspectives around the driving subject and process the original images of different perspectives to obtain multi-perspective features;

[0035] A difference module is used to perform difference processing on the multi-view features at the current moment and the multi-view features at the previous moment to obtain a time series difference feature containing time series information;

[0036] A variable depth extraction module, used to extract depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution;

[0037] An outer product operation module, used for performing an outer product operation on the variable depth distribution on the temporal difference feature to generate a frustum feature matrix;

[0038] A mapping module is used to map the visual cone feature matrix to a BEV space centered on the vehicle to generate an environmental perception result.

[0039] In addition, the present invention also proposes an environmental perception device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the pure visual autonomous driving environment perception method based on the variable depth mechanism as described above.

[0040] In addition, the present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the pure visual autonomous driving environment perception method based on the variable depth mechanism as described above is implemented.

[0041] The advantages of the present invention are:

[0042] (1) The present invention collects original images of different perspectives around the driving subject and processes the original images of different perspectives to obtain multi-perspective features. In order to improve the depth prediction accuracy, the multi-perspective features of the previous moment and the multi-perspective features of the current moment are differentiated to obtain time series difference features containing time series information; then, a variable depth mechanism is used to extract depth information from the time series difference features to obtain a variable depth distribution, a variable probability distribution of predicted objects at main depth values, so as to improve the prediction accuracy; and the variable depth distribution is combined with the time series difference features through an outer product operation to generate a cone feature matrix, which combines all possible combinations of input vectors, and each row shows a feature mapping at a specific depth; then, the cone feature matrix is ​​mapped to a BEV space centered on the vehicle; therefore, the present invention improves the accuracy of object depth estimation in image-based autonomous driving environment perception, provides environmental perception results for autonomous driving, and simplifies subsequent key tasks such as path planning.

[0043] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a method for pure visual autonomous driving environment perception based on a variable depth mechanism proposed in one embodiment of the present invention;

[0045] Figure 2 This is a block diagram of the implementation principle of a pure vision autonomous driving environment perception method based on a variable depth mechanism proposed in one embodiment of the present invention;

[0046] Figure 3 is a block diagram of the principle of variable depth distribution extraction in one embodiment of the present invention;

[0047] Figure 4 is a block diagram of the principle of generating a cone feature matrix in one embodiment of the present invention;

[0048] Figure 5 It is a structural schematic diagram of a pure visual autonomous driving environment perception system based on a variable depth mechanism proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] like Figure 1 to Figure 2 As shown, the first embodiment of the present invention proposes a pure visual autonomous driving environment perception method based on a variable depth mechanism, and the method includes the following steps:

[0051] S10, acquiring original images of different perspectives around the driving subject, and processing the original images of different perspectives to obtain multi-perspective features;

[0052] It should be noted that, in this embodiment, the autonomous driving vehicle can be equipped with cameras in six directions: front, left front, right front, left rear, right rear, and rear, and original images of different perspectives can be collected through cameras in different viewing directions.

[0053] S20, performing differential processing on the multi-view features at the current moment and the multi-view features at the previous moment to obtain a time series differential feature containing time series information;

[0054] S30, extracting depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution;

[0055] S40, performing an outer product operation on the variable depth distribution and the temporal difference feature to generate a frustum feature matrix;

[0056] S50: Map the visual cone feature matrix to the BEV space centered on the vehicle to generate an environmental perception result.

[0057] This embodiment collects original images of different perspectives around the driving subject and processes the original images of different perspectives to obtain multi-perspective features. In order to improve the depth prediction accuracy, the multi-perspective features of the previous moment and the multi-perspective features of the current moment are differentiated. Taking "ABC" and "BAC" as examples, the difference in the first position letters of ABC and BAC is calculated, taking A as an example, that is, AB, to obtain the time series difference feature containing time series information; then, the variable depth mechanism is used to extract the depth information of the time series difference feature to obtain a variable depth distribution, a variable probability distribution of the predicted object at the main depth value, so as to improve the prediction accuracy and improve the image-based automatic driving environment perception. The accuracy of object depth estimation is known; the variable depth distribution is combined with the temporal difference feature through the outer product operation to generate a cone feature matrix. The cone feature matrix combines all possible combinations of the input vectors, and each row shows the feature map at a specific depth; then the cone feature matrix is ​​mapped to the BEV space centered on the self-vehicle; by extracting information from each individual camera image, and then predicting the depth using a variable depth mechanism, the image is projected to the BEV space centered on the self-vehicle through projection mapping. This method usually involves building a three-dimensional spatial representation from each camera, which is usually called a bird's eye view (BEV), which is a bottom-up strategy to achieve information fusion from local to global. Through this visual conversion, this method provides a more comprehensive and accurate environmental perception for the autonomous driving system, further improving the safety and navigation accuracy of the vehicle's autonomous driving system in complex traffic scenarios.

[0058] As a further preferred technical solution, the step S10: processing the original images of different viewing angles to obtain multi-view features, comprises the following steps:

[0059] The backbone network is used to process the original images from different perspectives to obtain multi-scale features;

[0060] The multi-scale features are fused to obtain the multi-view features.

[0061] Specifically, this embodiment can use networks such as ResNet and EfficientNet to process the original images collected by each camera, obtain multi-scale features, and fuse the multi-scale features into multi-view features F t .

[0062] As a further preferred technical solution, the step S20: performing differential processing on the multi-view features at the current moment and the multi-view features at the previous moment to obtain a temporal differential feature containing temporal information, which is expressed as:

[0063]

[0064] In the formula, and They represent the multi-view features at time t and time t-1, N represents the number of viewpoints, Conv represents the convolution operation, T t Represents the temporal difference feature.

[0065] It should be noted that, from a visual perspective, objects at different distances have different changes between adjacent frames. To improve the accuracy of depth prediction, we propose a temporal feature difference method. The temporal feature difference strategy is based on the persistence and dynamic changes of objects and structures at consecutive time points. By analyzing the feature differences between adjacent moments, it enhances the understanding of scene depth. In the difference process, the convolution operation fuses the features after difference and strengthens the correlation of temporal features. By analyzing the feature differences of consecutive frames, the depth prediction accuracy is improved. In addition, differential feature fusion also improves the smoothness and reliability of environmental representation and reduces the errors caused by occlusion and sensor noise.

[0066] As a further preferred technical solution, the step S30: extracting depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution, comprises the following steps:

[0067] S31, using the first linear layer to process the temporal difference feature, and identifying M key depths where the object is most likely to exist;

[0068] S32, using a second linear layer to learn the probability distribution of objects at the M key depths;

[0069] S33. Fusing the M key depths with corresponding probability distributions respectively to obtain a variable depth distribution, where the variable depth distribution is a weighted distance distribution result.

[0070] Specifically, Figure 3 As shown in Figure 2, we first set the total number of variable depths to M, and then use the linear layer to process the temporal difference feature T. t , identifying the M key depths where the object is most likely to exist, such as H1, H2, and H3. Next, the model uses another linear layer to learn the probability distribution of the object at these M depths. After that, the M depth values ​​and the corresponding probabilities are fused to obtain a weighted distance distribution result D1, D2, and D3.

[0071] In terms of depth prediction, the traditional method needs to estimate the probability of each value in the continuous depth interval. This method is inefficient and impractical. Therefore, this embodiment proposes a variable depth prediction mechanism. By adopting this variable depth prediction method, more detailed predictions can be made on key depth values, thereby improving the accuracy of depth prediction and helping autonomous driving vehicles to understand the surrounding environment more effectively. In addition, this method reduces unnecessary predictions of unlikely depth values, thereby improving the efficiency of calculations and reducing the waste of computing resources.

[0072] As a further preferred technical solution, the step S40: performing an outer product operation on the variable depth distribution and the temporal difference feature to generate a frustum feature matrix, which is expressed as:

[0073] c d =c×d t

[0074] In the formula, c d is the cone feature matrix, c represents the feature vector of the temporal difference feature, d t represents the transpose of the depth vector d of the variable depth profile.

[0075] Specifically, Figure 4 As shown in Figure 1, after obtaining the depth vector d of the variable depth distribution of each point in the image feature and the feature vector c of the temporal difference feature, the outer product operation is used to fuse these two pieces of information to obtain the cone feature matrix, which combines all possible combinations of the input vectors. The generated cone feature c d is a C×D matrix, where C and D are the dimensions of c and d respectively. d Each column represents the dot product of an element in d and c, and each row shows the feature map at a specific depth.

[0076] As a further preferred technical solution, the step S50: mapping the visual cone feature matrix to the BEV space centered on the vehicle to generate an environmental perception result includes:

[0077] S51, mapping the feature points in the view cone feature matrix to the BEV space through the camera's intrinsic parameter matrix K and extrinsic parameter matrix R;

[0078] It should be noted that the spatial transformation process maps the feature points in the image space to the BEV space centered on the vehicle, that is, the cone feature c d The feature points in are mapped to the BEV space through the camera's intrinsic parameter matrix K and extrinsic parameter matrix R.

[0079] Specifically, the intrinsic parameter matrix K is first used to transform the feature points from normalized pixel coordinates (x, y) to camera coordinates (x c,y c , z c ), the conversion formula is as follows:

[0080] [x c ,y c , z c ] t =K -1 [x, y, 1] t

[0081] Then, the feature points are transformed from the camera coordinates (x c ,y c , z c ) to the vehicle coordinates (X, Y, Z), the conversion formula is as follows:

[0082] [X, Y, Z, 1] t =T -1 ·[x c ,y c , z c , 1] t

[0083] This series of transformations ensures that feature points can be accurately mapped to the BEV space.

[0084] S52, dividing the BEV space into grids, generating a cylinder for each grid, and assigning the feature points after spatial transformation to the nearest cylinder according to their positions in the BEV space;

[0085] S53: Perform average pooling on the feature points in each column to generate the environment perception result.

[0086] It should be noted that after the feature points are mapped to the BEV space, cylinder pooling is required to integrate them into the predetermined BEV grid. Specifically, a BEV space with a bottom size of H×W is divided into grids, and each BEV grid generates a cylinder. The spatially transformed feature points are assigned to the nearest cylinder according to their positions in the BEV space. After that, the feature points in each cylinder are average pooled to reduce the data dimension and extract representative features.

[0087] In this way, we finally get a BEV feature of size C×H×W, where C represents the dimension of the BEV feature, and H and W correspond to the bottom size of the BEV space. This feature integrates the information of the feature points in each BEV grid, provides environmental perception results for autonomous driving, and simplifies key tasks such as subsequent path planning.

[0088] This embodiment can utilize the image data captured by multiple cameras installed around the vehicle, process the image data through a variable depth and temporal feature difference mechanism, and generate a bird's-eye view (BEV) feature centered on the vehicle. Through this visual conversion, the method provides the autonomous driving system with a more comprehensive and accurate environmental perception, further improving the safety and navigation accuracy of the vehicle's autonomous driving system in complex traffic scenarios.

[0089] In the perception of autonomous driving environment, vehicle detection is extremely critical and affects the safety and stability of the system. In order to verify the effect of this method, a public multimodal autonomous driving dataset NuScenes was used. For the training loss indicator of the model, we selected the binary cross entropy loss function (BCELoss), and its mathematical expression is as follows:

[0090]

[0091] where y i is a binary label 0 or 1, p i Represents the probability that the model predicts that the sample is a positive class, and N is the number of samples in each batch. This loss function measures the accuracy of the model in environmental perception.

[0092] The BEVDot method proposed in this study only uses Figure 2 In the case of any five of the six cameras shown in the figure, namely, front, left front, right front, left rear, right rear, and rear, the Lift-Splat[1], RC-BEVFusion[2], and SOGDet-SE[3] methods are compared in the vehicle detection scenario. The results are shown in Table 1.

[0093] Table 1 Comparison of vehicle detection performance of different autonomous driving environment perception methods

[0094]

[0095] Where C stands for Camera, R stands for Radar, BCELoss stands for binary cross entropy loss, IoU stands for the intersection over union (IoU) of the predicted result and the true area, mAP stands for the average precision of the prediction, and NDS stands for the performance on the test indicators unique to the NuScenes dataset.

[0096] From the data in the table, we can see that compared with other methods, BEVDot performs well in vehicle detection tasks. Its BCELoss on the validation set is 0.152, showing its high accuracy in vehicle detection. In terms of the IoU indicator, BEVDot leads with a score of 0.387, which is significantly better than Lift-Splat's 0.242. Although the IoU of RC-BEVFusion and SOGDet-SE is not public, this achievement of BEVDot has fully demonstrated its advantages in spatial positioning. In terms of the mAP indicator, BEVDot leads with a score of 0.508. Although the NDS indicator of NuScenes is not involved in this study, BEVDot's outstanding performance in the mAP indicator has confirmed its effectiveness and robustness in vehicle detection.

[0097] The BEVDot method proposed in this embodiment is shown by a series of experimental results that the BEVDot method only uses Figure 2 In the case of any 5 of the 6 cameras shown in the figure, including the front, left front, right front, left rear, right rear, and rear, the method surpasses the existing methods such as Lift-Splat, RC-BEVFusion, and SOGDet-SE in terms of key performance indicators such as binary cross entropy loss (BCELoss), intersection over union (IoU), and mean average precision (mAP). The experimental results further show that this method effectively integrates information from multiple cameras and provides a more accurate environmental representation for the autonomous driving system.

[0098] like Figure 5 As shown, the second embodiment of the present invention proposes a pure visual autonomous driving environment perception system based on a variable depth mechanism, and the system includes:

[0099] The image acquisition module 10 is used to acquire original images of different perspectives around the driving subject, and process the original images of different perspectives to obtain multi-perspective features;

[0100] The difference module 20 is used to perform a difference process on the multi-view feature at the current moment and the multi-view feature at the previous moment to obtain a time series difference feature containing time series information;

[0101] A variable depth extraction module 30, configured to extract depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution;

[0102] An outer product operation module 40 is used to distribute the variable depth to the temporal difference feature to perform an outer product operation to generate a frustum feature matrix;

[0103] The mapping module 50 is used to map the visual cone feature matrix to the BEV space centered on the vehicle to generate an environmental perception result.

[0104] As a further preferred technical solution, the difference module 20 specifically uses the following formula to calculate the time series difference feature, which is expressed as:

[0105]

[0106] In the formula, and They represent the multi-view features at time t and time t-1, N represents the number of viewpoints, Conv represents the convolution operation, T t Represents the temporal difference feature.

[0107] As a further preferred technical solution, the variable depth extraction module 30 includes:

[0108] A depth recognition unit, configured to process the temporal difference feature using the first linear layer to identify M key depths where the object is most likely to exist;

[0109] A probability distribution learning unit, configured to learn the probability distribution of an object at the M key depths using a second linear layer;

[0110] A fusion unit is used to fuse the M key depths with corresponding probability distributions respectively to obtain a variable depth distribution, where the variable depth distribution is a weighted distance distribution result.

[0111] As a further preferred technical solution, the formula for generating the cone feature matrix by the outer product operation module 40 is expressed as:

[0112] c d =c×d T

[0113] In the formula, c d is the cone feature matrix, c represents the feature vector of the temporal difference feature, d t represents the transpose of the depth vector d of the variable depth profile.

[0114] As a further preferred technical solution, the mapping module 50 includes:

[0115] A mapping unit, used to map the feature points in the view cone feature matrix to the BEV space through the camera's intrinsic parameter matrix K and extrinsic parameter matrix R;

[0116] A grid division unit is used to divide the BEV space into grids, each grid generates a cylinder, and the feature points after spatial transformation are assigned to the nearest cylinder according to their positions in the BEV space;

[0117] The pooling unit is used to perform average pooling on the feature points in each column to generate the environment perception result.

[0118] It should be noted that other embodiments of the pure visual autonomous driving environment perception system based on the variable depth mechanism described in the present invention or implementation methods can refer to the above-mentioned method embodiments, which will not be repeated here.

[0119] In addition, the present invention also proposes an environmental perception device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the pure visual autonomous driving environment perception method based on the variable depth mechanism as described in the first embodiment above.

[0120] In addition, the present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the pure visual autonomous driving environment perception method based on a variable depth mechanism as described in the first embodiment above is implemented.

[0121] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0122] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0124] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0125] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A pure vision autonomous driving environment perception method based on a variable depth mechanism, characterized in that: The method comprises: Acquire original images of different perspectives around the driving subject, and process the original images of different perspectives to obtain multi-perspective features; Performing differential processing on the multi-view features at the current moment and the multi-view features at the previous moment to obtain a time series differential feature containing time series information; Using a variable depth mechanism to extract depth information from the temporal difference feature to obtain a variable depth distribution; Performing an outer product operation on the variable depth distribution and the temporal difference feature to generate a frustum feature matrix; The cone feature matrix is ​​mapped to the BEV space centered on the ego vehicle to generate an environmental perception result.

2. The pure visual autonomous driving environment perception method based on a variable depth mechanism according to claim 1, characterized in that: The processing of the original images at different viewing angles to obtain multi-view features includes: The backbone network is used to process the original images from different perspectives to obtain multi-scale features; The multi-scale features are fused to obtain the multi-view features.

3. The pure visual autonomous driving environment perception method based on variable depth mechanism according to claim 1, characterized in that: The multi-view features of the current moment are differentially processed with the multi-view features of the previous moment to obtain a temporal differential feature containing temporal information, which is expressed as follows: In the formula, and They represent the multi-view features at time t and time t-1, N represents the number of viewpoints, Conv represents the convolution operation, T t Represents the temporal difference feature.

4. The pure visual autonomous driving environment perception method based on a variable depth mechanism according to claim 1, characterized in that: The step of extracting depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution includes: Processing the temporal difference features using a first linear layer to identify M key depths where the object is most likely to exist; Using a second linear layer to learn the probability distribution of objects at the M key depths; The M key depths are respectively fused with the corresponding probability distributions to obtain a variable depth distribution, where the variable depth distribution is a weighted distance distribution result.

5. The pure visual autonomous driving environment perception method based on variable depth mechanism according to claim 1, characterized in that: The variable depth is distributed on the temporal difference feature to perform an outer product operation to generate a frustum feature matrix, which is expressed as follows: c d =c×d T In the formula, c d is the cone feature matrix, c represents the feature vector of the temporal difference feature, d T represents the transpose of the depth vector d of the variable depth profile.

6. The pure vision autonomous driving environment perception method based on variable depth mechanism according to claim 1, characterized in that: Mapping the cone feature matrix to the BEV space centered on the vehicle to generate an environmental perception result includes: Mapping the feature points in the cone feature matrix to the BEV space through the camera's intrinsic parameter matrix K and extrinsic parameter matrix R; The BEV space is divided into grids, each grid corresponds to a cylinder, and the feature points after spatial transformation are assigned to the nearest cylinder according to their positions in the BEV space; The feature points in each column are averaged and pooled to generate the environment perception result.

7. The pure vision autonomous driving environment perception method based on a variable depth mechanism according to any one of claims 1 to 6, characterized in that: The original images of different viewing angles are acquired through cameras of different viewing directions equipped by the driving subject.

8. A pure vision autonomous driving environment perception system based on a variable depth mechanism, characterized in that: The system comprises: An image acquisition module is used to acquire original images of different perspectives around the driving subject and process the original images of different perspectives to obtain multi-perspective features; A difference module is used to perform difference processing on the multi-view features at the current moment and the multi-view features at the previous moment to obtain a time series difference feature containing time series information; A variable depth extraction module, used to extract depth information from the temporal difference feature using a variable depth mechanism to obtain a variable depth distribution; An outer product operation module, used for distributing the variable depth to the temporal difference feature to perform an outer product operation to generate a frustum feature matrix; A mapping module is used to map the visual cone feature matrix to a BEV space centered on the vehicle to generate an environmental perception result.

9. An environment sensing device, characterized in that: The device includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Target identification method based on online depth feedback alignment

    CN119992479A