Environment perception method based on multi-resolution BEV

Abstract

The invention provides an environment perception method based on a multi-resolution BEV. The method comprises the following steps: acquiring a six-view image; constructing an environment perception prediction model based on the multi-resolution BEV; and inputting the six-view image into the multi-resolution BEV environment perception prediction model, respectively obtaining time features and space features for a BEV list containing a plurality of resolution BEVs, updating the high-quality BEV list based on the space features and the time features, fusing the features of each resolution BEV in the BEV list based on the BEV list, and generating a final BEV list for different environment perception tasks. According to the method, space-time feature extraction and fusion are carried out on the BEV list, time sequence information and high-level semantic information are fully utilized for enhancing environment perception, through design of the BEV list, more attention is paid to areas within 30 m, namely high-resolution BEV in the BEV list, and areas outside 30 m are represented by low-resolution BEV in the BEV list. According to the scheme, the high precision of the key area can be kept, and meanwhile the calculation overhead of the long-distance area is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision unmanned vehicle automatic driving, and relates to an environment perception method based on multi-resolution BEV. Background Art

[0002] With the rapid development of autonomous driving technology, environmental perception technology, as a crucial link, has received unprecedented attention. Given the high cost-effectiveness of cameras, camera-based environmental perception technology has gradually emerged and has become an emerging research direction in the field of computer vision. This technology mainly relies on images or video data captured by cameras to achieve recognition, understanding and prediction of the surrounding environment. Since cameras can record rich color and texture information and provide high-resolution images, they can efficiently and accurately identify various road traffic elements.

[0003] Especially in the unmanned vehicle inspection application scenario of Liquan Beer Factory, camera-based environmental perception technology plays a key role. Whether it is daily patrols inside the factory or security monitoring of specific areas, unmanned vehicles need to obtain environmental information in real time and make corresponding decisions. As an advanced technical means, the environmental perception method based on Bird's Eye View (BEV) converts the two-dimensional image data collected by the camera into the BEV space through geometric transformation, depth estimation and other algorithms, thereby extracting features from the image data and locating the target, which is crucial for the autonomous navigation of unmanned vehicles in complex environments.

[0004] However, the current BEV-based environmental perception solution uses a fixed-resolution BEV representation, which has unreasonable allocation of computing resources, which not only increases the energy consumption of the system, but also affects the efficiency and effectiveness of the unmanned vehicle in performing patrol tasks in the Liquan Beer Factory. Therefore, developing an efficient model that can ensure the detection accuracy of key areas and make rational use of computing resources is of great significance to improving the overall patrol performance of unmanned vehicles in the Liquan Beer Factory. The optimized environmental perception system not only helps to improve the patrol capability of unmanned vehicles, but also enhances their response speed to emergencies, providing technical support for achieving more intelligent and safe factory management. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes an environmental perception method based on multi-resolution BEV, which combines BEVs of different resolutions, uses the temporal attention module, spatial attention module and fusion self-attention module to enhance the expression characteristics of BEV, and finally applies it to different environmental perception tasks. Under the premise of ensuring the accuracy of key areas, the computing resources are reasonably used to efficiently and dynamically complete different environmental perception tasks.

[0006] The present invention provides an environment perception method based on multi-resolution BEV, comprising:

[0007] Acquire six-view images;

[0008] Construct an environmental perception model based on multi-resolution BEV;

[0009] The six-view images are input into the multi-resolution BEV environmental perception prediction model, and the temporal features and spatial features are respectively obtained for a BEV list containing BEVs of multiple resolutions. Based on the spatial features and the temporal features, a high-quality BEV list is updated. Based on the BEV list, the features of the BEVs of each resolution in the BEV list are integrated to generate a final BEVlist for different environmental perception tasks.

[0010] Optionally, the multi-resolution BEV-based environmental perception model includes: a multi-resolution BEV setting module, a feature extraction module, a temporal attention module, a spatial attention module, a fusion self-attention module and an environmental perception task head module; wherein the multi-resolution BEV setting module is: the input is a BEV list, which consists of a high-resolution BEV in the central area and four low-resolution BEVs in the surrounding areas.

[0011]

[0012] Optionally, the feature extraction module is:

[0013] Introduce the feature encoding module of Resnet101 network;

[0014] Optionally, the temporal attention module obtains temporal features including:

[0015] The BEV list obtained in the previous frame and the BEV list in the current frame are input into the temporal self-attention module, and the temporal features are extracted using the temporal attention module network.

[0016]

[0017] Q p is the BEV query vector at coordinate p = (x, y). Q is the BEV of the current frame, B t-1 is the BEV of the previous frame. Δp is the sum of Q and B t-1 Prediction. V is the value of the sampling point. Output BEV list of extracted time features.

[0018] Optionally, the spatial attention module acquires the spatial feature including:

[0019] The six-view image and the BEV list are input into the feature extraction module, and the spatial features are extracted using the spatial attention module network. p The z-axis is extended to form cylindrical query vectors, which are projected into six two-dimensional views. The hit view is called V pick Perform deformable attention on the two-dimensional query point (x′, y′) on the hit view, back-project it to the corresponding cylindrical vector, and finally perform weighted summation of all sampled features on the z-axis of the cylindrical vector to obtain the updated BEVlist as the output of spatial cross attention.

[0020]

[0021] Where N rp is the total 3D reference points for each BEV query, P is the projection function, i is the camera view index, j is the reference point index, and F t i is the feature of the i-th camera view.

[0022] Optionally, the self-attention fusion module fuses the self-attention including:

[0023] The BEV list output by the temporal and spatial attention modules is input into the fusion self-attention module to extract the self-attention features and obtain a high-quality BEV list for downstream tasks. p Mapping to the real world.

[0024] p rq =project(p q ) (4)

[0025] Among them, p q Q p The coordinates of rq For p q Mapped to real-world coordinates.

[0026] Calculate the offset:

[0027] Δp rqk =Lienear(Q) (5)

[0028] Δp rqk is p obtained by passing Q through a linear layer rq The offset in real-world coordinates.

[0029] Real-world sampling point coordinates:

[0030] p rk =p rq +Δp rqk(6)

[0031] p rk For p rq After the offset Δp rqk The real-world coordinates of the last sampling point k.

[0032] Real-world sample points are mapped to BEV coordinates:

[0033] p k =project(p rk ) (7)

[0034] p k For p rk The position (coordinates) projected into the BEV feature.

[0035]

[0036] W m A is a linear transformation of the output results of multiple heads. mqk =Lienear(Q),A mqk is the attention weight, which is obtained by directly passing the Q input feature through the linear layer. m F(p k ) is Value, where W m is the transformation matrix, F(p k )w is p k Characteristics of the location.

[0037] The set of updated Q is the BEV list of fused features.

[0038] Optionally, the environment perception task head module includes:

[0039] The BEV list output by the fused self-attention module is input into the occupancy prediction head and the point cloud segmentation head to output the results of different environmental perception tasks.

[0040] Therefore, the present invention adopts the above-mentioned environment perception method based on multi-resolution BEV, which has the following beneficial effects:

[0041] (1) The present invention uses a BEV list consisting of a high-resolution BEV in a central area and four low-resolution BEVs in surrounding areas to replace the fixed-resolution BEV expression features, so that the model can better utilize computing resources while maintaining the detection accuracy of key areas.

[0042] (2) The present invention adopts the fusion self-attention mechanism to efficiently integrate the features of high-resolution BEV and low-resolution BEV in the BEV list as well as the spatial features, so that the model can better combine the spatial information and improve the detection accuracy of high-resolution and low-resolution boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 For the framework diagram. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] It should be noted that the steps illustrated in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0047] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0048] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0049] As shown in the figure, the present invention proposes an environment perception method based on multi-resolution BEV, including:

[0050] Acquire six-view images;

[0051] Construct an environmental perception model based on multi-resolution BEV;

[0052] The six-view images are input into the multi-resolution BEV environmental perception prediction model, and the temporal features and spatial features are respectively obtained for a BEV list containing BEVs of multiple resolutions. Based on the spatial features and the temporal features, a high-quality BEV list is updated. Based on the BEV list, the features of the BEVs of each resolution in the BEV list are integrated to generate a final BEVlist for different environmental perception tasks.

[0053] Furthermore, the environmental perception model based on multi-resolution BEV includes: a multi-resolution BEV setting module, a feature extraction module, a temporal attention module, a spatial attention module, a fused self-attention module, and an environmental perception task head module; wherein the multi-resolution BEV setting module is: the input is a BEV list, which consists of a high-resolution BEV in the central area and N low-resolution BEVs in the surrounding areas.

[0054] Furthermore, the feature extraction module is: a feature encoding module that introduces the Resnet101 network;

[0055] Furthermore, the temporal features acquired by the temporal attention module include:

[0056] The BEV list obtained in the previous frame is input into the temporal self-attention module, and the temporal features are extracted using the temporal attention module network. The BEV list with extracted temporal features is output.

[0057] Furthermore, the spatial attention module obtains the spatial features including:

[0058] The six-view images and the BEV list are input into the feature extraction module, and the spatial features are extracted using the spatial attention module network.

[0059] Furthermore, the fusion self-attention module fusion self-attention includes:

[0060] The BEV list output by the temporal and spatial attention modules is input into the fusion self-attention module to extract the self-attention features to obtain a high-quality BEV list for downstream tasks.

[0061] Furthermore, the BEV list output by the fused self-attention module is input into the occupancy prediction head and the point cloud segmentation head to output the results of different environmental perception tasks.

[0062] Example

[0063] As shown in the figure, the present embodiment provides an environment perception method based on multi-resolution BEV, which specifically includes the following steps:

[0064] Step 1: Acquire six-view images around the motor vehicle;

[0065] The six-view images surrounding the motor vehicle contain almost all the information in the current scene, which is a prerequisite for completing the environmental perception task.

[0066] Step 2: Multi-resolution BEV-based environmental perception model;

[0067] In order to make better use of six-view images, the model introduces Resnet101 in the feature extraction module to extract the feature information of the six-view images. In addition, the model also designs a multi-resolution BEV setting module, a temporal attention module, a spatial attention module, a fused self-attention module, and an environmental perception task head module.

[0068] Step 3: Set the multi-resolution BEV to a BEV list consisting of a central area high-resolution BEV and N surrounding area low-resolution BEVs.

[0069]

[0070] Step 4: Substitute the BEV list obtained by the model in the previous frame and the BEV list of the current frame into the temporal attention module. The BEV list of the current frame fully extracts the feature information of the previous frame to obtain the temporal features and enhance the changing motion features.

[0071]

[0072] Q p is the BEV query vector at coordinate p = (x, y). Q is the BEV of the current frame, B t-1 is the BEV of the previous frame. Δp is the sum of Q and B t-1 Prediction. V is the value of the sampling point. Output BEV list of extracted time features.

[0073] Step 5: Input the six-view image obtained in step 2 and the BEV list obtained in step 4 into the spatial attention module. p The z-axis is extended to form cylindrical query vectors, which are projected into six two-dimensional views. The hit view is called V pickPerform deformable attention on the two-dimensional query point (x′, y′) on the hit view, back-project it to the corresponding cylindrical vector, and finally perform weighted summation of all sampled features on the z-axis of the cylindrical vector to obtain the updated BEV list as the output of spatial cross attention.

[0074]

[0075] Where N rp is the total 3D reference points for each BEV query, P is the projection function, i is the camera view index, j is the reference point index, and F t i is the feature of the i-th camera view.

[0076] Step 6: Input the BEVlist obtained in step 5 into the fusion self-attention module to fuse features between different resolutions in the BEVlist to obtain a high-quality BEVlist for downstream tasks. p Mapping to the real world.

[0077] p rq =project(p q ) (4)

[0078] Among them, p q Q p The coordinates of rq For p q Mapped to real-world coordinates.

[0079] Calculate the offset:

[0080] Δp rqk =Lienear(Q) (5)

[0081] Δp rqk is p obtained by passing Q through a linear layer rq Coordinate offset in the real world. Real world sampling point coordinates:

[0082] p rk =p rq +Δp rqk (6)

[0083] p rk For p rq After the offset Δp rqk The real-world coordinates of the last sampling point k.

[0084] Real world sample points are mapped to BEV coordinates:

[0085] p k =project(p rk) (7)

[0086] p k For p rk The position (coordinates) projected into the BEV feature.

[0087]

[0088] W m A is a linear transformation of the output results of multiple heads. mqk =Lienear(Q),A mqk is the attention weight, which is obtained by directly passing the Q input feature through the linear layer. m F(p k ) is Value, where W m is the transformation matrix, F(p k )w is p k Characteristics of the location.

[0089] The set of updated Q is the BEVlist of fused features.

[0090] Step 7: Input the BEVlist obtained in step 6 into the environment perception task head module, namely the occupancy prediction head and the point cloud segmentation head, and output the results of different environment perception tasks.

[0091] As an improvement to the BEV-based environment perception model, this embodiment proposes an environment perception model based on multi-resolution BEV. Among them, the six-view image features are extracted through ResNet101, the features of the previous frame are fully utilized for temporal feature fusion, and the spatial features are fully acquired by sampling the six-view image features. Then, the features between multi-resolution BEVs are fused into the final BEVlist features, and the features are applied to different environment perception tasks, thereby improving the accuracy of environment perception.

[0092] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be covered within the protection scope of the present application.

[0093] Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for environmental perception based on multi-resolution BEV, characterized in that: include: Acquire six-view images; Construct an environmental perception model based on multi-resolution BEV; The six-view images are input into the multi-resolution BEV environmental perception prediction model, and the temporal features and spatial features are respectively obtained for a BEVlist containing multiple resolution BEVs. Based on the spatial features and temporal features, the BEVlist is updated. Based on the BEV list, the features of the BEVs of each resolution in the BEVlist are integrated to generate a final BEVlist for different environmental perception tasks.

2. The method for environmental perception based on multi-resolution BEV according to claim 1, characterized in that: The multi-resolution BEV-based environmental perception model includes: a multi-resolution BEV setting module, a feature extraction module, a temporal attention module, a spatial attention module, a fused self-attention module and an environmental perception task head module; wherein, the multi-resolution BEV setting module is: the input is a BEVlist, which consists of a high-resolution BEV in the central area and N low-resolution BEVs in the surrounding areas.

3. The feature extraction module according to claim 2, characterized in that: The feature encoding module of the Resnet101 network is introduced to complete feature extraction.

4. The method for environmental perception based on multi-resolution BEV according to claim 2, characterized in that: The temporal features acquired by the temporal attention module include: The BEVlist obtained from the previous frame and the BEVlist of the current frame are input into the temporal self-attention module, and the temporal features are extracted using the temporal attention module network. Q p is the BEV query vector at coordinate p = (x, y). Q is the BEV of the current frame, B t-1 is the BEV of the previous frame. Δp is the sum of Q and B t-1 Prediction. V is the value of the sampling point. Output BEVlist of extracted time features.

5. The multi-resolution BEV-based environment perception method according to claim 2, characterized in that: Spatial attention acquires the spatial features including: The six-view image and BEVlist are input into the feature extraction module, and the spatial features are extracted using the spatial attention module network. p The z-axis is extended to form cylindrical query vectors, which are projected into six two-dimensional views. The hit view is called V pick Perform deformable attention on the two-dimensional query point (x′, y′) on the hit view, back-project it to the corresponding cylindrical vector, and finally perform weighted summation of all sampled features on the z-axis of the cylindrical vector to obtain the updated BEVlist as the output of spatial cross attention. Where N rp is the total 3D reference points for each BEV query, P is the projection function, i is the camera view index, j is the reference point index, and F t i is the feature of the i-th camera view.

6. The method for environmental perception based on multi-resolution BEV according to claim 2, characterized in that: The fusion self-attention module includes: The BEV list output by the temporal and spatial attention modules is input into the fusion self-attention module to extract the self-attention features and obtain a high-quality BEV list for downstream tasks. p Mapping to the real world. p rq =project(p q ) (4) Among them, p q Q p The coordinates of rq For p q Mapped to real-world coordinates. Calculate the offset: Δp rqk =Lienear(Q) (5) Δp rqk is p obtained by passing Q through a linear layer rq Coordinate offset in the real world. Real world sampling point coordinates: p rk =p rq +Δp rqk (6) p rk For p rq After the offset Δp rqk The real-world coordinates of the last sampling point k. Real-world sample points are mapped to BEV coordinates: p k =project(p rk ) (7) p k For p rk The position (coordinates) projected into the BEV feature. W m A is a linear transformation of the output results of multiple heads. mqk =Lienear(Q),A mqk is the attention weight, which is obtained by directly passing the Q input feature through the linear layer. m F(p k ) is Value, where W m is the transformation matrix, F(p k )w is p k Characteristics of the location. The set of updated Q is the BEVlist of fused features.

7. The multi-resolution BEV-based environment perception method according to claim 2, characterized in that: The environment perception task head module includes: The BEV list output by the fused self-attention module is input into the occupancy prediction head and the point cloud segmentation head to output the results of different environmental perception tasks.

Images