Environment perception method and system of vehicle and training method of environment perception model

By combining multi-stage fusion processing of image sensors and multiple laser sensors on autonomous vehicles, detailed environmental perception information is generated, which solves the problem of low perception accuracy of the vehicle in complex environments and light changes, and improves the close-range and Blind spot recognition ability.

CN119131734APending Publication Date: 2024-12-13杭州吉利汽车有限公司 +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411297803.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-12-13

AI Technical Summary

Technical Problem

The existing environmental perception technology of autonomous driving vehicles has low accuracy in complex environments and light changes, especially inadequate perception of close and blind spots.

Method used

By installing an image sensor, a first laser sensor and a second laser sensor on the vehicle, and using an attention mechanism network to perform multi-stage fusion processing of the acquired image features, bird's-eye point cloud features and distance point cloud features, it generates texture information, Depth information and height information target environment perception information.

Benefits of technology

It significantly improves the adaptability of autonomous vehicles to complex environments and light changes, improves the ability to identify targets such as "ghost probes" and low obstacles, enhances the ability to perceive close-range and blind spots, and solves the accuracy of vehicle environment perception. Low problem.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119131734A_ABST
    Figure CN119131734A_ABST
Patent Text Reader

Abstract

The invention relates to an environment perception method and system of a vehicle and a training method of an environment perception model.The environment perception method of the vehicle comprises the steps that image features carrying texture information are obtained based on an image sensor on the vehicle; acquiring aerial view point cloud features carrying depth information based on a first laser sensor; based on a second laser sensor, distance point cloud features carrying height information are obtained; the mounting positions of the first laser sensor and the second laser sensor on the vehicle are different; and carrying out fusion processing on the image features, the bird's-eye view point cloud features and the distance point cloud features to generate target environment perception information including texture information, depth information and texture information. According to the invention, the problem of low accuracy of vehicle environment perception in a complex scene is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a vehicle's environmental perception method, system, and training method for an environmental perception model. Background Art

[0002] Autonomous driving perception technology is a very important module for driverless cars. Perception mainly includes tasks such as target detection, segmentation, depth estimation and prediction, which can help vehicles perceive road environment information through sensors. However, in related technologies, environmental perception methods mainly rely on a single sensor, such as lidar or camera, which often find it difficult to provide comprehensive and accurate environmental information in complex environments. For example, in low light or bad weather, the perception ability of the camera will be greatly reduced; and although lidar is not affected by light, it may be difficult to distinguish different objects in some complex scenes, thus affecting the accuracy of vehicle environmental perception.

[0003] Currently, no effective solution has been proposed for the problem of low accuracy of vehicle environment perception in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a vehicle environment perception method, system and environment perception model training method to at least solve the problem of low accuracy of vehicle environment perception in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a method for environmental perception of a vehicle, the method comprising:

[0006] Based on the image sensor on the vehicle, acquiring image features carrying texture information;

[0007] Based on the first laser sensor, a bird's-eye view point cloud feature carrying depth information is obtained;

[0008] Based on the second laser sensor, a distance point cloud feature carrying height information is obtained; the first laser sensor and the second laser sensor are installed at different positions on the vehicle;

[0009] The image features, the bird's-eye view point cloud features and the distance point cloud features are fused to generate target environment perception information including the texture information, the depth information and the texture information.

[0010] In some embodiments, the fusing of the image features, the bird's-eye view point cloud features and the distance point cloud features to generate target environment perception information further includes:

[0011] Performing feature fusion on the image feature and the bird's-eye view point cloud feature to obtain a first fused feature;

[0012] Performing feature fusion on the bird's-eye view point cloud feature and the distance point cloud feature to obtain a second fused feature;

[0013] The first fusion feature and the second fusion feature are fused to obtain the target environment perception information.

[0014] In some embodiments, obtaining the first fusion feature, the second fusion feature, and the target environment perception information includes:

[0015] Inputting the image features and the bird's-eye view point cloud features into a first attention mechanism network for weight processing to obtain a weight score, and performing feature fusion on the image features and the bird's-eye view point cloud features based on the weight score to obtain the first fused feature;

[0016] Inputting the bird's-eye view point cloud feature and the distance point cloud feature into a second attention mechanism network for feature fusion to obtain the second fused feature;

[0017] The first fusion feature and the second fusion feature are input into a third attention mechanism network for feature fusion to obtain the target environment perception information; wherein the first attention mechanism network and the second attention mechanism network are respectively connected to the third attention mechanism network.

[0018] In some embodiments, obtaining the image features, the bird's-eye view point cloud features, and the distance point cloud features further includes:

[0019] Inputting the image information collected by the image sensor into the first encoder model for feature extraction processing to obtain the image features; the first encoder model is connected to the first attention mechanism network;

[0020] Inputting the first point cloud data collected by the first laser sensor into the second encoder model for feature extraction processing to obtain the bird's-eye view point cloud features; the second encoder model is respectively connected to the first attention mechanism network and the second attention mechanism network;

[0021] The second point cloud data collected by the second laser sensor is input into the third encoder model for feature extraction processing to obtain the distance point cloud feature; the third encoder model is connected to the second attention mechanism network.

[0022] In some embodiments, the performing feature fusion on the first fusion feature and the second fusion feature to obtain the target environment perception information further includes:

[0023] Inputting the first fused feature and the second fused feature into a third attention mechanism network for feature fusion to obtain a target fused feature;

[0024] The target fusion feature is input into the decoder model for decoding processing, and the target environment perception information is output; wherein the third attention mechanism network is connected to the decoder model.

[0025] In some embodiments, the first laser sensor is disposed on the top of the vehicle, and the second laser sensor is disposed on the front of the vehicle; and / or, there are at least two image sensors, each of which constitutes a surround view image sensor.

[0026] In a second aspect, an embodiment of the present application provides a method for training an environment perception model, the method comprising:

[0027] Obtaining a training data set; the training data set carries a real environment label;

[0028] Inputting the training data set into a first initial encoder, a second initial encoder and a third initial encoder respectively, and outputting corresponding first training features, second training features and third training features;

[0029] Inputting the first training feature and the second training feature into a first initial attention mechanism network, and outputting a fourth training feature;

[0030] Input the second training feature and the third training feature into a second initial attention mechanism network, and output a fifth training feature;

[0031] Inputting the fourth training feature and the fifth training feature into the third initial attention mechanism network, and outputting a training fusion feature;

[0032] Inputting the training fusion features into a first initial decoder, and outputting a first prediction result;

[0033] According to the first prediction result and the real environment label, a first loss function result is calculated, and the gradient of the first loss function result is back-transferred to the first initial decoder for iterative training to generate a trained environment perception model.

[0034] In some embodiments, the step of back-transferring the gradient of the first loss function result to the first initial decoder for iterative training to generate a trained environment perception model includes:

[0035] Inputting the fourth training feature into a second initial decoder and outputting a second prediction result; calculating a second loss function result according to the second prediction result and the real environment label;

[0036] Input the fifth training feature into a third initial decoder, and output a third prediction result; calculate a third loss function result according to the third prediction result and the real environment label;

[0037] Calculate a total loss function result according to the first loss function result, the second loss function result and the third loss function result;

[0038] The gradient of the total loss function result is back-transferred for iterative training to generate the environment perception model.

[0039] In a third aspect, an embodiment of the present application provides an environment perception system for a vehicle, including an image sensor, a first laser sensor, a second laser sensor, and a controller;

[0040] The controller is respectively connected to the image sensor, the first laser sensor, and the second laser sensor, and is used to execute the vehicle environment perception method as described in the first aspect above.

[0041] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising the vehicle environment perception system as described in the second aspect above.

[0042] Compared with the related art, the vehicle environment perception method, system and training method of the environment perception model provided in the embodiment of the present application obtain image features carrying texture information based on the image sensor on the vehicle; obtain bird's-eye view point cloud features carrying depth information based on the first laser sensor; obtain distance point cloud features carrying height information based on the second laser sensor; the first laser sensor and the second laser sensor are installed at different positions on the vehicle; the image features, bird's-eye view point cloud features and distance point cloud features are fused and processed to generate target environment perception information including texture information, depth information and texture information. Based on this, accurate three-dimensional target detection can be performed on the surrounding environment of the autonomous driving vehicle, effectively improving the recognition ability of targets such as "ghost heads" and "low obstacles", significantly improving the perception ability of close distances and blind spots in complex environments, and having a strong adaptability to complex environments and lighting changes, thereby solving the problem of low accuracy of vehicle environment perception in complex scenes.

[0043] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 is a hardware structure block diagram of a terminal of a vehicle environment perception method according to an embodiment of the present application;

[0046] Figure 2 is a flow chart of a vehicle environment perception method according to an embodiment of the present application;

[0047] Figure 3 is a flow chart of another vehicle environment perception method according to an embodiment of the present application;

[0048] Figure 4 is a schematic diagram of an environment perception method for a vehicle according to an embodiment of the present application;

[0049] Figure 5 It is a structural block diagram of an environmental perception system for a vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0051] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0052] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0053] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 1 is a hardware structure block diagram of a terminal of a vehicle environment perception method according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0054] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the vehicle environment perception method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0056] This embodiment provides a vehicle environment perception method. Figure 2 is a flow chart of a vehicle environment perception method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0057] Step S210: acquiring image features carrying texture information based on an image sensor on the vehicle.

[0058] Image sensors (such as cameras) can capture visual information around the vehicle, including color, texture, shape, etc. These image data are converted into feature representations through image processing algorithms (such as edge detection, feature extraction, etc.). These features contain rich texture information and help identify objects such as road signs, pedestrians, and vehicles.

[0059] Step S220: Acquire bird's-eye view point cloud features carrying depth information based on the first laser sensor.

[0060] The first laser sensor (such as laser radar) is usually installed on the top or at a higher position of the vehicle to obtain a wider field of view and a longer detection distance. During the data acquisition process, the first laser sensor measures the distance between the objects in the surrounding environment and the vehicle (i.e., depth information) by continuously rotating and emitting laser beams, and generates three-dimensional point cloud data. These data points contain the three-dimensional coordinates (X, Y, Z) of each measured object. Point cloud data is usually stored as a numpy array with N rows and at least 3 columns, where N is the number of reflection points of the laser beam, each row corresponds to a separate point, and at least 3 values ​​(X, Y, Z) are used to represent the position of the point in space. In addition, additional values ​​for each point may be included, such as reflectivity (r), forming an N*4 matrix. Since the coordinate axes of the image and the coordinate axes of the point cloud are different, the point cloud data needs to be converted from three-dimensional space to a two-dimensional plane to generate a bird's-eye view. Usually, the X and Y coordinates of the point cloud (i.e., the position on the horizontal plane) are mapped to the pixel position of the image, while the Z coordinate (height information) is ignored. Next, feature extraction is performed on the above data to obtain the bird's eye view point cloud features in the bird's eye view (BEV) space.

[0061] Step S230: Based on the second laser sensor, a distance point cloud feature carrying height information is obtained; the first laser sensor and the second laser sensor are installed at different positions on the vehicle.

[0062] Since the first laser sensor at a position such as the top of the vehicle has a blind spot for close-range detection, it is difficult to identify short-range low obstacles, such as stone piers in the park, fallen bicycles on the road, garbage, accident debris and other obstacles, which can easily affect the safety of autonomous driving. Therefore, in order to improve the above problems, in this step, the second point cloud data collected by the second laser sensor is also obtained. Among them, the second laser sensor can be arranged at a lower position in front of the vehicle, such as the front bumper of the vehicle; this type of position helps to reduce blind spots and better detect obstacles near the ground. The second laser sensor works in a similar manner to the first laser sensor, measuring the distance between objects and the vehicle in the surrounding environment (i.e., depth information) by emitting a laser beam and receiving a reflected signal. These data are also recorded in the form of a three-dimensional point cloud. Due to the low installation position, the second laser sensor can capture more detailed information about obstacles near the ground, including their height, shape and position. From the point cloud data collected by the second laser sensor, special attention is paid to the height information (Z coordinate) of each point. Various low obstacles such as stone piers, fallen bicycles, garbage piles, accident debris, etc. can be identified by using the height information and other point cloud features (such as point cloud density, shape, etc.). The distance point cloud features extracted in the above way are in the range view (RV) space.

[0063] Step S240, fusing the image features, the bird's-eye view point cloud features and the distance point cloud features to generate target environment perception information including texture information, depth information and texture information.

[0064] In this step, a more comprehensive and accurate environmental perception model can be generated by fusing data from different sensors. Specifically, image features provide rich texture and color information, which helps to identify object categories; bird's-eye view point cloud features provide depth information of the environment and layout information from a bird's-eye view; and distance point cloud features supplement the detailed three-dimensional structure and height information of the objects in front. By fusing this information together, the system can understand the surrounding environment more accurately and provide strong support for subsequent decision-making and planning.

[0065] In the environmental perception method of the above-mentioned vehicle, by fusing the features of the image sensor and the laser sensors at different positions, accurate three-dimensional target detection can be performed on the environment around the autonomous driving vehicle, effectively improving the recognition ability of targets such as "ghost heads" and "low obstacles", and significantly improving the perception ability of close distances and blind spots in complex environments. It has a strong adaptability to complex environments and lighting changes, and solves the problem of low accuracy of vehicle environmental perception.

[0066] In some embodiments, a vehicle environment perception method is provided. Figure 3 is a flow chart of another vehicle environment perception method according to an embodiment of the present application. Figure 3 As shown, the process includes Figure 2 The steps S210 to S230 shown in the figure further include the following steps:

[0067] Step S310: performing feature fusion on the image features and the bird's-eye view point cloud features to obtain a first fused feature.

[0068] Among them, the bird's-eye view point cloud features extracted by the first laser sensor collect data and provide the true value information of the depth, and the image features extracted by the image sensor collect data and provide semantic and texture information. The data is then spliced ​​in the feature dimension for the two subsequent features, so that the first fused feature contains these two types of information, so that each feature point in the image can be located in combination with the depth coordinates provided by the bird's-eye view point cloud features, which helps the system to more accurately understand the objects in the environment and their positional relationships, and overcomes the shortcomings of inaccurate depth information in pure vision solutions. It should be supplemented that after the above-mentioned feature splicing process, the first fused feature obtained is a feature in the BEV space.

[0069] In an optional embodiment, the acquisition process of the above-mentioned first fusion feature may also include the following steps: inputting the image features and the bird's-eye view point cloud features into the first attention mechanism network for weight processing to obtain a weight score, and performing feature fusion on the image features and the bird's-eye view point cloud features based on the weight score to obtain the first fusion feature. Specifically, in order to further improve the fusion effect, the image features and the bird's-eye view point cloud features can be input into the first attention mechanism network (such as a channel attention mechanism network). The network assigns weights to each channel by learning the importance of different channel features, and then performs weighted processing on the features based on these weights. Finally, the weighted features are spliced ​​to obtain the first fusion feature. This method can make the fusion process more intelligent and flexible, which is conducive to highlighting key information and suppressing noise and redundancy. By introducing the attention mechanism to splice the image features and the bird's-eye view point cloud features, it is beneficial to improve the accuracy of feature fusion. In particular, the first attention mechanism network can be a channel attention mechanism network.

[0070] Step S320, performing feature fusion on the bird's-eye view point cloud features and the distance point cloud features to obtain a second fused feature.

[0071] Among them, the fusion features of the laser sensor in the BEV space and RV space are obtained by fusion of the bird's-eye view point cloud features and the distance point cloud features. For example, the coordinates of each point in the BEV space are expressed as (x, y, z0), where z0 is a fixed z-axis coordinate in the space; the coordinates of each point in the RV space are expressed as (x0', y', z'), where x0' is a fixed x-axis coordinate in the space. After the above fusion processing, the second fusion feature of the known x-coordinate and z-coordinate can be obtained, so that it has both the information in the BEV space and the information in the RV in the height space, which overcomes the difficulty of the pure BEV solution in detecting multiple objects in the height space.

[0072] In an optional embodiment, the process of obtaining the above-mentioned second fused feature may also include the following steps: inputting the bird's-eye view point cloud feature and the distance point cloud feature into the second attention mechanism network for feature fusion to obtain the second fused feature. In the second attention mechanism network, the attention weights of the two are calculated by calculating the cosine similarity between the bird's-eye view feature and the distance view feature, and the bird's-eye view feature and the distance view feature are weighted fused according to the calculated attention weights, so that the elements in each feature will obtain different weights according to their similarity with the elements in other features, thereby realizing effective integration of information. After weighted fusion of the attention mechanism, the network will output the second fused feature; this feature contains both the extensive spatial information of the bird's-eye view and the depth and dynamic information of the distance view, so as to better describe complex scenes and targets. The second attention mechanism network can be a cross attention mechanism network.

[0073] Step S330: Fusing the first fusion feature and the second fusion feature to obtain target environment perception information.

[0074] In this step, the first fusion feature and the second fusion feature are fused in the second fusion stage to obtain the final feature, which is then decoded to generate the above-mentioned target environment perception information. In an optional embodiment, the further fusion process of the above-mentioned first fusion feature and the second fusion feature may also include the following steps: inputting the first fusion feature and the second fusion feature into the third attention mechanism network for feature fusion to obtain the target environment perception information; wherein the first attention mechanism network and the second attention mechanism network are respectively connected to the third attention mechanism network. Specifically, the first attention mechanism network and the second attention mechanism network are respectively used as pre-processing steps, and their outputs (i.e., the first fusion feature and the second fusion feature) are sent to the third attention mechanism network. In the third attention mechanism network, the attention weight between the first fusion feature and the second fusion feature is first calculated, and the first fusion feature and the second fusion feature are weightedly fused according to the calculated attention weight; this process effectively integrates and reorganizes the information in the two features to generate more comprehensive and accurate target environment perception information. In particular, the third attention mechanism network can be a cross attention mechanism network.

[0075] Through the above steps S310 to S330, a multi-stage fusion mechanism is introduced. Compared with the method of directly fusing various features, the fusion accuracy is better and the feature information can be better reflected, which is beneficial to improving the accuracy of environmental fusion perception.

[0076] In some embodiments, the above process of obtaining image features, bird's-eye view point cloud features, and distance point cloud features may further include the following steps:

[0077] The image information collected by the image sensor is input into the first encoder model for feature extraction and outputting the image features obtained after encoding. The first encoder model is connected to the first attention mechanism network. The image information collected by the image sensor is input into the first encoder model. This encoder model is usually a convolutional neural network (CNN) or its variants, such as a residual network (ResNet), a VGG (Visual Geometry Group) network, etc., in order to extract high-level feature representations from the image. The first encoder model processes the input image layer by layer, and gradually extracts the key features in the image through structures such as convolutional layers, pooling layers, and activation functions. Finally, after encoding, the first encoder model outputs the encoded image features. These features contain rich information in the image, providing a basis for subsequent feature fusion and decision-making.

[0078] The first point cloud data collected by the first laser sensor is input into the second encoder model for feature extraction processing to obtain the bird's-eye view point cloud features; the second encoder model is connected to the first attention mechanism network and the second attention mechanism network respectively. The first point cloud data collected by the first laser sensor is input into the second encoder model. This encoder model may be designed for point cloud data, such as Point Cloud Net (PointNet), PointNet++, etc., which can process disordered point cloud data and extract useful features. The second encoder model processes the input point cloud data and extracts features such as spatial layout and depth information under the bird's-eye view through operations such as point cloud convolution and pooling. The bird's-eye view point cloud features output by the second encoder model are not only used for subsequent feature splicing processing, but also connected to the first attention mechanism network and the second attention mechanism network respectively. In the first attention mechanism network, the bird's-eye view point cloud features will interact with the image features to optimize the weight distribution of the image features; in the second attention mechanism network, the bird's-eye view point cloud features will interact with the distance point cloud features to optimize the fusion effect of the two.

[0079] The second point cloud data collected by the second laser sensor is input into the third encoder model for feature extraction and processing to obtain the distance point cloud features; the third encoder model is connected to the second attention mechanism network. The second point cloud data collected by the second laser sensor is input into the third encoder model. This encoder model is similar to the second encoder model and is also designed for point cloud data to extract features under the distance field of view. The third encoder model processes the input point cloud data and extracts features related to distance, speed, etc. These features help understand the dynamic characteristics of objects in the environment. Connecting the attention mechanism network: The distance point cloud features output by the third encoder model are connected to the second attention mechanism network. In the second attention mechanism network, the distance point cloud features interact with the bird's-eye view point cloud features to further optimize the fusion effect of the two.

[0080] In some embodiments, the above-mentioned feature fusion of the first fusion feature and the second fusion feature to obtain the target environment perception information further includes the following steps:

[0081] The first fused feature and the second fused feature are input into the third attention mechanism network for feature fusion to obtain the target fused feature; the target fused feature is input into the decoder model for decoding processing, and the target environment perception information is output; wherein the third attention mechanism network is connected to the decoder model. Specifically, after the third attention mechanism network performs feature fusion processing on the first fused feature and the second fused feature, the target fused feature is output. The target fused feature is used as the input of the decoder model. The task of the decoder model is to convert this high-level feature representation back to the original space or a higher-level abstract representation to generate the target environment perception information. More specifically, the decoder model is usually composed of multiple layers, including but not limited to feature extractors, upsampling layers, convolutional layers, batch normalization layers (BN layers), and activation layers (such as ReLU layers). Each network layer works together to decode the encoded feature map to restore or improve the resolution and expression ability of the feature map. In the decoding process, the decoder first receives the output from the previous network, and the feature extractor (such as the convolutional layer) in the decoder further processes the received feature map to extract higher-level features. These feature extraction layers usually keep the resolution of the feature map unchanged, but may change the number of channels of the feature map. In order to restore the resolution of the feature map, the decoder uses an upsampling layer. Upsampling can be achieved through a variety of methods, such as linear interpolation, deconvolution (transposed convolution), etc. These methods can increase the size of the feature map to make it closer to the resolution of the original input data. After feature extraction and upsampling, the decoder usually uses activation layers such as Rectified Linear Unit (ReLU) to perform nonlinear transformations on the feature map to enhance the expressiveness of the model. Finally, the decoder outputs decoded feature maps, which contain the information required for environmental perception. Depending on the specific task, the output feature map may need to be further processed (such as classification, regression, etc.) to obtain the final environmental perception result. For example, in an autonomous driving system, the decoder can decode the features of roads, vehicles, pedestrians, etc. extracted by the encoder to restore a complete road scene map, thereby providing the vehicle with accurate environmental perception information.

[0082] The following is an explanation of the environmental perception process of the above vehicle in conjunction with a specific embodiment. Take the vehicle in this embodiment where the perception sensors include: 6 surround view cameras, 1 first laser radar installed on the roof, and 1 second laser radar located on the front bumper of the vehicle as an example. Figure 4 , the process includes the following steps:

[0083] First, in the network input stage, the surround camera composed of 6 cameras can collect 6 images at different angles around the vehicle at the current moment, with the pixel format of [red, green, blue]; it covers the 360-degree scene around the vehicle, and each image is used as the network input, which is encoded by the first encoder model and outputs the image features in the BEV space. At the same time, a frame of laser point cloud is collected by the first laser radar, in the format of [x, y, z, intensity]; it is input into the second encoder model for encoding and outputting the bird's-eye view point cloud features in the BEV space, and a frame of point cloud data is collected by the second laser radar, which is input into the third encoder model for encoding and outputting the distance point cloud features in the RV space.

[0084] Next, the above-mentioned features are fused in stages in the feature fusion stage. Specifically, in the first fusion stage, the image features and the bird's-eye view point cloud features are used as the input of the channel attention mechanism network for feature splicing to obtain the first fusion feature of the lidar and the camera in the BEV space; the bird's-eye view point cloud features and the distance point cloud features are used as the input of the cross attention mechanism network 1 for feature splicing to obtain the second fusion feature that integrates the information in the BEV space and the information in the RV space. In the second fusion stage, the first fusion feature and the second fusion feature are fused through the cross attention mechanism 2 to obtain the target fusion feature.

[0085] The fused features can be output through the decoding operation of the decoder. Taking pedestrians, vehicles, and bicycles as examples, the environmental perception results output after decoding include the above three types of targets; of course, the actual situation can be expanded according to different businesses. Finally, the above method realizes the autonomous driving environment perception method based on multimodal input, which can accurately perceive targets in close range or blind spots in different application scenarios such as logistics vehicles and passenger cars, effectively improving the safety of autonomous driving.

[0086] In some of the embodiments, a training method for an environment perception model including the above-mentioned multiple network structures is also provided, which includes the following steps:

[0087] Obtain a training data set; the training data set carries real environment labels. The training data can be generated by collecting test data of various sensors in a test environment, or by using historical data, and carries labels such as pedestrians, vehicles, or roads that correspond to business needs.

[0088] The training data set is input into the first initial encoder, the second initial encoder and the third initial encoder respectively, and the corresponding first training features, second training features and third training features are output; the first training features and the second training features are input into the first initial attention mechanism network, and the fourth training features are output; the second training features and the third training features are input into the second initial attention mechanism network, and the fifth training features are output; the fourth training features and the fifth training features are input into the third initial attention mechanism network, and the training fusion features are output; the training fusion features are input into the first initial decoder, and the first prediction result is output.

[0089] According to the first prediction result and the real environment label, use an appropriate loss function (such as cross entropy loss, mean square error loss or intersection over union (IoU) loss, etc., depending on the task type) to calculate the loss value and obtain the first loss function result, which can be expressed as Total Loss; and pass the gradient of the first loss function result back to the initial decoder for iterative training to update the network parameters in each model. Repeat the above training process until the number of iterative training is met or the encoder model converges, and finally generate a trained environment perception model. The model structure of the environment perception model can be found in Figure 4 , including a trained first encoder model, a second encoder model, a third encoder model, a first attention mechanism network, a second attention mechanism network, a third attention mechanism network and an encoder model.

[0090] Through the above-mentioned embodiments, the model is iteratively trained by calculating various loss function values, which is beneficial to improving the accuracy of the output data of each model and further improving the accuracy of vehicle environment perception.

[0091] The calculation of the loss function and the iterative training process also include the following steps:

[0092] The fourth training feature is input into the second initial decoder, and the second prediction result is output; the second loss function result is calculated based on the second prediction result and the real environment label; the second loss function result can be expressed as BEV loss. The fifth training feature is input into the third initial decoder, and the third prediction result is output; the third loss function result is calculated based on the third prediction result and the real environment label; the third loss function result can be expressed as RV loss. It should be noted that the above-mentioned second initial decoder and the above-mentioned third initial decoder only play a role in the training stage to supervise the network.

[0093] According to the first loss function result, the second loss function result and the third loss function result, the total loss function result is calculated. The gradient of the total loss function result is back-transferred for iterative training to generate an environmental perception model including a first encoder model, a second encoder model, a third encoder model, a first attention mechanism network, a second attention mechanism network, a third attention mechanism network and an encoder model. Specifically, the above three loss function values ​​are added together, and a calculation graph is automatically generated, and then each layer of the network is updated according to the calculation graph. Among them, the gradient before each layer is updated is obtained by calculating and summing all the outputs separately. For example, the first encoder model and the first attention mechanism network use BEV loss+Total Los to update the parameters, the second encoder model uses BEV loss+RV loss+Total Los to update the parameters, the third encoder model and the second attention mechanism network use RV loss+Total Loss to update, and the third attention mechanism network and the decoder use Total Loss to update the network parameters. It can be understood that in this iterative training process, the network parameters of the second initial decoder model and the third initial decoder model will also be updated together to improve the accuracy of network training.

[0094] The training method provided by the above embodiment not only ensures the effectiveness of network training, but also avoids the need for complex post-processing for target fusion after the first fusion feature and the second fusion feature are output separately, thereby ensuring the simplicity and efficiency of the forward reasoning target output.

[0095] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0096] This embodiment also provides a vehicle environment perception system, Figure 5 is a structural block diagram of an environment perception system for a vehicle according to an embodiment of the present application, such as Figure 5 As shown, the system includes: an image sensor 51, a first laser sensor 52, a second laser sensor 53 and a controller 54. The controller 54 is connected to the image sensor 51, the first laser sensor 52 and the second laser sensor 53 respectively, and is used to execute the vehicle environment perception method as described in any of the above embodiments.

[0097] This embodiment also provides a vehicle, which includes the above-mentioned vehicle environment perception system.

[0098] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0099] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0100] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0101] S1, based on the image sensor on the vehicle, obtain image features carrying texture information.

[0102] S2, based on the first laser sensor, obtaining bird's-eye view point cloud features carrying depth information.

[0103] S3, based on the second laser sensor, obtaining distance point cloud features carrying height information; the first laser sensor and the second laser sensor are installed at different positions on the vehicle.

[0104] S4, fusing the image features, the bird's-eye view point cloud features and the distance point cloud features to generate target environment perception information including texture information, depth information and texture information.

[0105] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0106] In addition, in combination with the vehicle environment perception method in the above embodiments, the present application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any vehicle environment perception method in the above embodiments is implemented.

[0107] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A vehicle environment perception method, characterized in that: The method comprises: Based on the image sensor on the vehicle, acquiring image features carrying texture information; Based on the first laser sensor, a bird's-eye view point cloud feature carrying depth information is obtained; Based on the second laser sensor, a distance point cloud feature carrying height information is obtained; the first laser sensor and the second laser sensor are installed at different positions on the vehicle; The image features, the bird's-eye view point cloud features and the distance point cloud features are fused to generate target environment perception information including the texture information, the depth information and the texture information.

2. The environment perception method according to claim 1, characterized in that: The fusing process of the image features, the bird's-eye view point cloud features and the distance point cloud features to generate target environment perception information further includes: Fusing the image feature and the bird's-eye view point cloud feature to obtain a first fused feature; Performing feature fusion on the bird's-eye view point cloud feature and the distance point cloud feature to obtain a second fused feature; The first fusion feature and the second fusion feature are fused to obtain the target environment perception information.

3. The environment perception method according to claim 2, characterized in that: Acquiring the first fusion feature, the second fusion feature, and the target environment perception information includes: Inputting the image features and the bird's-eye view point cloud features into a first attention mechanism network for weight processing to obtain a weight score, and performing feature fusion on the image features and the bird's-eye view point cloud features based on the weight score to obtain the first fused feature; Inputting the bird's-eye view point cloud feature and the distance point cloud feature into a second attention mechanism network for feature fusion to obtain the second fused feature; The first fusion feature and the second fusion feature are input into a third attention mechanism network for feature fusion to obtain the target environment perception information; wherein the first attention mechanism network and the second attention mechanism network are respectively connected to the third attention mechanism network.

4. The environment perception method according to claim 3, characterized in that: Acquiring the image features, the bird's-eye view point cloud features and the distance point cloud features also includes: Inputting the image information collected by the image sensor into the first encoder model for feature extraction processing to obtain the image features; the first encoder model is connected to the first attention mechanism network; Inputting the first point cloud data collected by the first laser sensor into the second encoder model for feature extraction processing to obtain the bird's-eye view point cloud features; the second encoder model is respectively connected to the first attention mechanism network and the second attention mechanism network; The second point cloud data collected by the second laser sensor is input into the third encoder model for feature extraction processing to obtain the distance point cloud feature; the third encoder model is connected to the second attention mechanism network.

5. The environment perception method according to claim 4, characterized in that: The performing feature fusion on the first fusion feature and the second fusion feature to obtain the target environment perception information further includes: Inputting the first fused feature and the second fused feature into a third attention mechanism network for feature fusion to obtain a target fused feature; The target fusion feature is input into the decoder model for decoding processing, and the target environment perception information is output; wherein the third attention mechanism network is connected to the decoder model.

6. The environment perception method according to any one of claims 1 to 5, characterized in that: The first laser sensor is arranged on the top of the vehicle, and the second laser sensor is arranged on the front of the vehicle; and / or, there are at least two image sensors, and each of the image sensors constitutes a surround view image sensor.

7. A method for training an environment perception model, characterized in that: The method comprises: Obtaining a training data set; the training data set carries a real environment label; Inputting the training data set into a first initial encoder, a second initial encoder and a third initial encoder respectively, and outputting corresponding first training features, second training features and third training features; Inputting the first training feature and the second training feature into a first initial attention mechanism network, and outputting a fourth training feature; Input the second training feature and the third training feature into a second initial attention mechanism network, and output a fifth training feature; Inputting the fourth training feature and the fifth training feature into the third initial attention mechanism network, and outputting a training fusion feature; Inputting the training fusion features into a first initial decoder, and outputting a first prediction result; According to the first prediction result and the real environment label, a first loss function result is calculated, and the gradient of the first loss function result is back-transferred to the first initial decoder for iterative training to generate a trained environment perception model.

8. The training method according to claim 7, characterized in that: The step of back-transferring the gradient of the first loss function result to the first initial decoder for iterative training to generate a trained environment perception model includes: Inputting the fourth training feature into a second initial decoder and outputting a second prediction result; calculating a second loss function result according to the second prediction result and the real environment label; Input the fifth training feature into a third initial decoder, and output a third prediction result; calculate a third loss function result according to the third prediction result and the real environment label; Calculate a total loss function result according to the first loss function result, the second loss function result and the third loss function result; The gradient of the total loss function result is back-transferred for iterative training to generate the environment perception model.

9. A vehicle environment perception system, characterized in that: comprising an image sensor, a first laser sensor, a second laser sensor and a controller; The controller is respectively connected to the image sensor, the first laser sensor, and the second laser sensor, and is used to execute the vehicle environment perception method as described in any one of claims 1 to 6.

10. A vehicle, characterized in that: An environmental perception system for a vehicle including the one described in claim 9.

Citation Information

Cited By

  • Feature fusion method and device, environment perception method and device, equipment, medium and product

    CN120852941A

  • Environment sensing method, system and equipment based on vehicle sentry mode and medium

    CN121716646A