Integrated environmental perception system for intelligent connected vehicles

Abstract

The present invention relates to the field of automobile environment perception technology, and specifically refers to a comprehensive environment perception system for intelligent networked vehicles, which specifically includes: a subject vehicle, an object vehicle, and an environment perception module. In view of the problem of lack of diversity in information granularity when capturing environmental feature information, the present invention proposes a multi-perspective comprehensive perception model based on a multi-level attention encoder, divides image features into different regions using a multi-scale window segmentation method, and learns each region through sparse attention, and further cascades learning of output features of different granularities through multiple coding branches of different depths, thereby enhancing the adaptability and robustness of the system in complex traffic scenarios; in view of the problem of spatiotemporal misalignment of features when intelligent networked vehicles share features, time alignment coding is proposed, and accurate timestamp information is embedded in the shared feature data of different intelligent networked vehicles, so that the feature data received by each vehicle can accurately reflect the real-time environmental status.

Description

Technical Field

[0001] The present invention relates to the field of automobile environment perception technology, and specifically to a comprehensive environment perception system for intelligent networked vehicles. Background Art

[0002] The comprehensive environmental perception system of intelligent connected vehicles refers to a system that uses a variety of sensors in intelligent connected vehicles to perceive the surrounding environment, and processes and analyzes the perceived information to fully understand the vehicle's driving environment. Intelligent connected vehicles refer to vehicles that are deeply integrated with new-generation information and communication technologies, artificial intelligence technologies, and Internet technologies. In the development of intelligent connected vehicles, the comprehensive environmental perception system plays a vital role. In order to ensure driving safety and improve the driving experience, intelligent connected vehicles need to have comprehensive perception capabilities of the surrounding environment.

[0003] However, the existing integrated environmental perception system of intelligent connected vehicles has the following problems:

[0004] The information granularity and diversity of intelligent connected vehicles when capturing environmental feature information are insufficient, making it difficult to conduct comprehensive perception;

[0005] When intelligent connected vehicles share features, time delays cause temporal and spatial misalignment of features, resulting in inconsistent perception of the environment by different vehicles in the same scene. Summary of the invention

[0006] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a comprehensive environmental perception system for intelligent connected vehicles. In view of the problem that the information granularity diversity is insufficient when intelligent connected vehicles capture environmental feature information, which makes it difficult to comprehensively perceive it, the present invention proposes a multi-perspective comprehensive perception model based on a multi-level attention encoder, and adopts a multi-level attention encoder in the model for feature learning. First, the features of the surround image of the intelligent connected vehicle are divided into different regions using a multi-scale window segmentation method, and each region is learned through sparse attention, and then cascade learning is performed through multiple encoding branches of different depths, so that the model can effectively capture the relationship between the global and the local, which not only optimizes the vehicle's perception of the surrounding The environmental perception accuracy also enhances the adaptability and robustness of the system in complex traffic scenarios, providing more reliable environmental perception support for intelligent connected vehicles. In view of the problem that when intelligent connected vehicles share features, the spatiotemporal misalignment of features caused by time delay makes different vehicles in the same scenario have inconsistent perception of the environment, the present invention proposes time alignment coding, embeds accurate timestamp information into the shared feature data of different intelligent connected vehicles, and effectively solves the time inconsistency of feature information between different vehicles by time marking and synchronous processing of feature data, so that the feature data received by each vehicle can accurately reflect the real-time environmental status, enhancing the overall robustness of the environmental perception system of the intelligent connected vehicle.

[0007] The technical solution adopted by the present invention is as follows: The present invention provides a comprehensive environment perception system for intelligent networked vehicles, specifically comprising: a subject vehicle, an object vehicle, and an environment perception module, wherein the subject vehicle and the object vehicle are deployed with an on-board intelligent networked system, and the environment perception module is deployed in the on-board intelligent networked system;

[0008] The object car is another intelligent network-connected car in the same area as the subject car. The subject car and the object car respectively collect images of the front, rear, left front, left rear, right front and right rear of the car body through six cameras, and use these six images as surround view images;

[0009] The environment perception module uses a multi-view comprehensive perception model based on a multi-level attention encoder to learn the features of the surround images of the subject car and the object car through the multi-level attention encoder, and further realizes feature sharing and fusion;

[0010] The multi-view comprehensive perception model based on the multi-level attention encoder specifically includes: a subject car feature extraction unit, an object car feature extraction unit, a feature fusion unit and a multi-task learning unit.

[0011] The subject vehicle feature extraction unit extracts features of T groups of temporally continuous surround view images of the subject vehicle through a ResNet34 model, and learns the features of the surround view images through a multi-level attention encoder to obtain comprehensive features of the subject vehicle;

[0012] The object car feature extraction unit extracts features of T groups of time-continuous surround view images of the object car through a ResNet34 model, learns the features of the surround view images through a multi-level attention encoder, and obtains comprehensive features of the object car;

[0013] The multi-level attention encoder of the subject vehicle feature extraction unit and the multi-level attention encoder of the object vehicle feature extraction unit share weights.

[0014] The multi-level attention encoder is composed of N encoding branches of different depths, where The number of layers of the encoding branch is , the process of learning the features of surround images through a multi-level attention encoder is as follows:

[0015] Step D1: Use the multi-scale window segmentation method to obtain the segmentation tensor set of each encoding branch: The multi-scale window segmentation method segments the features of the surround image through a multi-scale window. The specific content is: define the size coefficient, and let the size of the feature of the surround image be ,in Indicates height, Indicates width, represents the total number of channels, then The size of the multi-scale window of the encoding branch is ,in is the size coefficient, and the multi-scale window of each encoding branch is used to segment the surround image features to obtain a set of segmentation tensors, namely ,in Indicates The first coding branch A partition tensor, T is the number of feature groups of the time-continuous surround image;

[0016] Step D2: The first layer of each encoding branch calculates the sparse attention of each segmentation tensor in the segmentation tensor set, and then concatenates the sparse attention results of each segmentation tensor to obtain the output of the first layer of each encoding branch. The specific formula is as follows:

[0017] ;

[0018] ;

[0019] ;

[0020] in, For the The first coding branch A partition tensor, is the index subscript of the partitioned tensor, , and Respectively The query vector, key vector and value vector of , and Respectively , and The weight matrix of Indicates The first coding branch The output of the segmented tensor, Indicates The first coding branch The output of the segmented tensor, represents sparse attention, Indicates The first layer output of the encoding branch;

[0021] Step D3: Use cross attention to calculate the The output of the layer, Greater than or equal to 2: The first The output of the layer is used as the query vector. The output of the layer is used as the key vector and the value vector to calculate the cross attention. The specific formula is as follows:

[0022] ;

[0023] in, Indicates The first coding branch The output of the layer, Indicates The first coding branch The output of the layer, Indicates The first coding branch The output of the layer, represents the softmax function, express The query vector is express The key vector of express The value vector of DRepresents dimension;

[0024] Step D4: Take the Nth layer output of the Nth encoding branch as the final output of the multi-level attention encoder.

[0025] The feature fusion unit shares the comprehensive features of the subject car and the comprehensive features of the object car, and fuses them with the time alignment coding. The specific steps are as follows:

[0026] Step S1: compressing and transmitting the comprehensive features of the object car to the subject car;

[0027] Step S2: After receiving the compressed comprehensive features of the object car, the subject car decompresses the features and combines the comprehensive features of the object car with the comprehensive features of the subject car to obtain fusion features;

[0028] Step S3: construct the time alignment coding of the fusion features. The calculation formula of the time alignment coding is as follows:

[0029] ;

[0030] ;

[0031] in, and They represent the first The first tensor and Time-aligned coding of channels, is the index of the channel, and represent the sine function and cosine function respectively, The first The time delay of a tensor;

[0032] Step S4: Perform a linear transformation on the time alignment code, add the linear transformation result to the fusion feature, and obtain the aligned feature. The specific formula is as follows:

[0033] ;

[0034] in, represents the fusion feature, represents the aligned features, represents a linear transformation, Represents time-aligned coding.

[0035] The multi-task learning unit performs multi-task learning on the aligned features, and the multi-tasks specifically include 3D target detection, motion trajectory prediction, and bird's-eye view perception.

[0036] The beneficial results achieved by the present invention using the above scheme are as follows:

[0037] (1) To address the problem that intelligent connected vehicles have insufficient information granularity diversity when capturing environmental feature information, which makes it difficult to conduct comprehensive perception, the present invention proposes a multi-view comprehensive perception model based on a multi-level attention encoder, and adopts a multi-level attention encoder for feature learning in the model. First, a multi-scale window segmentation method is used to divide the features of the surround view image of the intelligent connected vehicle into different regions, and sparse attention is used to learn each region. Then, cascade learning is performed through multiple encoding branches of different depths, so that the model can effectively capture the relationship between the global and the local. This not only optimizes the vehicle's perception accuracy of the surrounding environment, but also enhances the adaptability and robustness of the system in complex traffic scenarios, providing more reliable environmental perception support for intelligent connected vehicles.

[0038] (2) To address the problem that when intelligent connected vehicles share features, the spatiotemporal misalignment of features caused by time delay leads to inconsistency in the environmental perception of different vehicles in the same scene. The present invention proposes time alignment coding to embed accurate timestamp information into the shared feature data of different intelligent connected vehicles. By time-marking and synchronizing the feature data, the time inconsistency of feature information between different vehicles is effectively resolved, so that the feature data received by each vehicle can accurately reflect the real-time environmental status, thereby enhancing the overall robustness of the intelligent connected vehicle environmental perception system. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is the architecture diagram of the multi-view comprehensive perception model based on the multi-level attention encoder proposed in the present invention;

[0040] Figure 2 This is the architecture diagram of the multi-level attention encoder proposed in the present invention;

[0041] Figure 3 This is a schematic diagram of the multi-scale window segmentation method proposed in the present invention.

[0042] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] Example 1, see Figure 1 , Figure 1 This is an architecture diagram of a multi-view comprehensive perception model based on a multi-level attention encoder proposed in the present invention. In this embodiment, a comprehensive environment perception system for intelligent connected vehicles is proposed. The specific contents are as follows:

[0045] The present invention provides a comprehensive environment perception system for intelligent networked vehicles, specifically comprising: a subject vehicle, an object vehicle, and an environment perception module, wherein the subject vehicle and the object vehicle are deployed with an on-board intelligent networked system, and the environment perception module is deployed in the on-board intelligent networked system;

[0046] The object car is another intelligent connected car in the same area as the subject car, providing auxiliary information for the subject car's perception and decision-making. The subject car and the object car use six cameras to collect images of the front, rear, left front, left rear, right front and right rear of the car body respectively, and use these six images as surround view images;

[0047] The environment perception module uses a multi-view comprehensive perception model based on a multi-level attention encoder to learn the features of the surround images of the subject car and the object car through the multi-level attention encoder, and further realizes feature sharing and fusion;

[0048] The multi-view comprehensive perception model based on the multi-level attention encoder specifically includes: a subject car feature extraction unit, an object car feature extraction unit, a feature fusion unit and a multi-task learning unit.

[0049] In this embodiment, the object vehicles in the same area are determined to be all the intelligent connected vehicles other than the subject vehicle in a 70m×70m area centered on the subject vehicle.

[0050] The six cameras in this embodiment cover the 360-degree environment around the vehicle body without blind spots.

[0051] In this embodiment, the construction and training of the multi-view comprehensive perception model based on the multi-level attention encoder is implemented in Python 3.7 language, using the PyTorch deep learning framework, and the GPU model is NVIDIA GeForce RTX3090.

[0052] Embodiment 2: Based on the above embodiment, this embodiment implements a subject vehicle feature extraction unit and an object vehicle feature extraction unit. The specific contents are as follows:

[0053] The subject car feature extraction unit extracts the features of T groups of temporally continuous surround images of the subject car through the ResNet34 model, and learns the features of the surround images through a multi-level attention encoder to obtain the comprehensive features of the subject car;

[0054] The object car feature extraction unit extracts the features of T groups of time-continuous surround view images of the object car through the ResNet34 model, and learns the features of the surround view images through a multi-level attention encoder to obtain the comprehensive features of the object car;

[0055] The multi-level attention encoder of the subject car feature extraction unit and the multi-level attention encoder of the object car feature extraction unit share weights.

[0056] The surround view image in this embodiment is an RGB image with a resolution of 800×600. The feature size of each surround view image extracted by the ResNet34 model is 64×64, and the number of channels is 128.

[0057] Example 3, see Figure 2 and Figure 3 , Figure 2 This is the architecture diagram of the multi-level attention encoder proposed in this invention. Figure 3 This is a schematic diagram of the multi-scale window segmentation method proposed in the present invention. Based on the above embodiment, this embodiment implements a multi-level attention encoder. The specific contents are as follows:

[0058] The multi-level attention encoder consists of N encoding branches of different depths, where The number of layers of the encoding branch is , the number of encoding branches in this embodiment is 8, and the process of learning the features of the surround image through the multi-level attention encoder is as follows:

[0059] Step D1: Use the multi-scale window segmentation method to obtain the segmentation tensor set of each encoding branch: The multi-scale window segmentation method segments the features of the surround image through a multi-scale window. The specific content is: define the size coefficient, and let the size of the feature of the surround image be ,in Indicates height, Indicates width, represents the total number of channels, then The size of the multi-scale window of the encoding branch is ,in is the size coefficient, and the multi-scale window of each encoding branch is used to segment the surround image features to obtain a set of segmentation tensors, namely ,in Indicates The first coding branch A partition tensor, T is the number of feature groups of the time-continuous surround image;

[0060] In this embodiment, the size of the feature of the surround image is , the size coefficient is set to 4, then the sizes of the multi-scale windows of the 8 coding branches are , , , , , , , ,When segmenting through multi-scale windows, the tensor that is less than the window size is filled with pixels through the boundary reflection filling method;

[0061] Step D2: Construct the key vector, value vector and query vector of each segmentation tensor in the segmentation tensor set of each encoding branch. The first layer of each encoding branch calculates the sparse attention of each segmentation tensor, and then concatenates the sparse attention results of each segmentation tensor. The size of the concatenated features is consistent with the size of the features of the surround image, and the output of the first layer of each encoding branch is obtained. The specific formula is as follows:

[0062] ;

[0063] ;

[0064] ;

[0065] in, For the The first coding branch A partition tensor, is the index subscript of the partitioned tensor, , and Respectively The query vector, key vector and value vector of , and Respectively , and The weight matrix of Indicates The first coding branch The output of the segmented tensor, Indicates The first coding branch The output of the segmented tensor, represents sparse attention, Indicates The first layer output of the encoding branch;

[0066] Step D3: Use cross attention to calculate the The output of the layer, Greater than or equal to 2: The first The output of the layer is used as the query vector. The output of the layer is used as the key vector and the value vector to calculate the cross attention. The specific formula is as follows:

[0067] ;

[0068] in, Indicates The first coding branch The output of the layer, Indicates The first coding branch The output of the layer, Indicates The first coding branch The output of the layer, represents the softmax function, express The query vector is express The key vector of express The value vector of D Represents dimension;

[0069] Step D4: Take the Nth layer output of the Nth encoding branch as the final output of the multi-level attention encoder.

[0070] Embodiment 4: Based on the above embodiment, this embodiment implements a feature fusion unit, and the specific contents are as follows:

[0071] The feature fusion unit shares the comprehensive features of the subject car and the comprehensive features of the object car, and fuses them with the time alignment coding. The specific steps are as follows:

[0072] Step S1: compress and transmit the comprehensive features of the object car to the subject car. In this embodiment, the compression algorithm is specifically the LZMA compression algorithm;

[0073] Step S2: After receiving the compressed comprehensive features of the object car, the subject car decompresses the features and combines the comprehensive features of the object car with the comprehensive features of the subject car to obtain fusion features;

[0074] Step S3: construct the time alignment coding of the fusion features. The calculation formula of the time alignment coding is as follows:

[0075] ;

[0076] ;

[0077] in, and They represent the first The tensor and Time-aligned coding of channels, is the index of the channel, and represent the sine function and cosine function respectively, The first The time delay of a tensor;

[0078] No. The time delay of a tensor is specifically the difference between the timestamp of the surround view image of the subject car and the timestamp of the surround view image of the object car corresponding to the tensor;

[0079] Step S4: Perform a linear transformation on the time alignment code, add the linear transformation result to the fusion feature, and obtain the aligned feature. The specific formula is as follows:

[0080] ;

[0081] in, represents the fusion feature, represents the aligned features, represents a linear transformation, Represents time-aligned coding.

Claims

1. The integrated environment perception system of intelligent connected vehicles is characterized by: Specifically include: A subject car, an object car, and an environment perception module, wherein the subject car and the object car are deployed with an on-board intelligent network connection system, and the environment perception module is deployed in the on-board intelligent network connection system; The subject car and the object car collect surround images; The environment perception module uses a multi-view comprehensive perception model based on a multi-level attention encoder to learn the features of the surround images of the subject car and the object car through the multi-level attention encoder to achieve feature sharing and fusion; The multi-view comprehensive perception model based on the multi-level attention encoder specifically includes: a subject car feature extraction unit, an object car feature extraction unit, a feature fusion unit and a multi-task learning unit; The multi-level attention encoder is composed of N encoding branches of different depths, where The number of layers of the encoding branch is , the process of learning the features of surround images through a multi-level attention encoder is as follows: Step D1: using a multi-scale window segmentation method to obtain a set of segmentation tensors for each encoding branch; Step D2: The first layer of each encoding branch calculates the sparse attention of each segmentation tensor in the segmentation tensor set, and then concatenates the sparse attention results of each segmentation tensor to obtain the output of the first layer of each encoding branch; Step D3: Use cross attention to calculate the The output of the layer, Greater than or equal to 2: The first The output of the layer is used as the query vector. The output of the layer is used as the key vector and value vector to calculate the cross attention; Step D4: Take the Nth layer output of the Nth encoding branch as the final output of the multi-level attention encoder.

2. The integrated environment perception system for intelligent connected vehicles according to claim 1, characterized in that: The subject vehicle feature extraction unit extracts features of T groups of temporally continuous surround view images of the subject vehicle through a ResNet34 model, and learns the features of the surround view images through a multi-level attention encoder to obtain comprehensive features of the subject vehicle; The object car feature extraction unit extracts features of T groups of time-continuous surround images of the object car through a ResNet34 model, learns the features of the surround images through a multi-level attention encoder, and obtains comprehensive features of the object car.

3. The integrated environment perception system for intelligent connected vehicles according to claim 2, characterized in that: The multi-level attention encoder of the subject vehicle feature extraction unit and the multi-level attention encoder of the object vehicle feature extraction unit share weights.

4. The integrated environment perception system for intelligent connected vehicles according to claim 2, characterized in that: The specific content of the multi-scale window segmentation method is: segment the features of the surround image through the multi-scale window: define the size coefficient, and let the size of the feature of the surround image be ,in Indicates height, Indicates width, represents the total number of channels, then The size of the multi-scale window of the encoding branch is ,in is the size coefficient, and the multi-scale window of each encoding branch is used to segment the surround image features to obtain a set of segmentation tensors. T is the number of feature groups of the time-continuous surround image.

5. The integrated environment perception system for intelligent connected vehicles according to claim 4, characterized in that: The feature fusion unit shares the comprehensive features of the subject car and the comprehensive features of the object car, constructs time alignment coding and fuses them. The specific steps are as follows: Step S1: compressing and transmitting the comprehensive features of the object car to the subject car; Step S2: After receiving the compressed comprehensive features of the object car, the subject car decompresses the features and combines the comprehensive features of the object car with the comprehensive features of the subject car to obtain fusion features; Step S3: constructing a time-aligned encoding of fused features; Step S4: Perform a linear transformation on the time-aligned code, and add the linear transformation result to the fusion feature.

6. The integrated environment perception system for intelligent connected vehicles according to claim 5, characterized in that: The calculation formula of the time alignment coding is as follows: ； ； in, and They represent the first The first tensor and Time alignment coding of channels, is the index of the channel, and represent the sine function and cosine function respectively, The first The time delay of a tensor.

7. The integrated environment perception system for intelligent connected vehicles according to claim 1, characterized in that: The object car is another intelligent connected car in the same area as the subject car. The subject car and the object car respectively collect images of the front, rear, left front, left rear, right front and right rear of the car body through six cameras, and use these six images as surround view images.

Images