Obstacle recognition method, system, equipment, product and medium for multi-train vehicles

Abstract

The present invention discloses an obstacle recognition method, system, equipment, product and medium for multi-unit vehicles, and relates to the field of transportation. In order to solve the problem of high cost of obstacle perception through laser radar, the method includes obtaining sub-surround view images collected by multiple cameras on the vehicle for each vehicle, extracting two-dimensional image features from all sub-surround view images, processing the camera parameters, bird's-eye view coding and two-dimensional image features corresponding to the camera through a single-carriage bird's-eye view converter, and obtaining the current bird's-eye view features of the vehicle; fusing the current bird's-eye view features of multiple vehicles to obtain multi-carriage fusion features; and obtaining obstacle information of multi-unit vehicles using multi-carriage fusion features. The present invention can reduce the cost of obstacle perception and improve the reliability and comprehensiveness of obstacle perception.

Description

Technical Field

[0001] The present invention relates to the field of transportation, and in particular to an obstacle recognition method, system, equipment, product and medium for multi-train vehicles. Background Art

[0002] The multi-unit articulated vehicle is a new type of urban passenger bus that moves on rubber wheels without fixed tracks. Compared with ordinary buses, it can transport more passengers by connecting multiple carriages through an articulated plate, alleviating the pressure on the public transportation system. Since the multi-unit articulated vehicle uses an articulated plate to connect multiple groups of carriages, the vehicle body is long, and it is generally difficult for the driver to detect obstacles around the vehicle body with the naked eye. At present, multi-unit articulated vehicles mainly use laser radar for obstacle perception, but laser radar has problems such as high price and short service life.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the invention

[0004] The purpose of the present invention is to provide an obstacle recognition method, system, device, product and medium for multi-train vehicles, which can reduce the obstacle perception cost and improve the reliability and comprehensiveness of obstacle perception.

[0005] On the one hand, the present invention provides an obstacle recognition method for a multi-unit vehicle, wherein the multi-unit vehicle comprises a plurality of articulated carriages, each of which is provided with a plurality of cameras, the cameras being used to collect sub-surround view images within their own collection areas, and the obstacle recognition method for the multi-unit vehicle comprises:

[0006] For each of the carriages, the sub-surround view images collected by the multiple cameras on the carriage are obtained, two-dimensional image features are extracted from all the sub-surround view images, and the camera parameters, bird's-eye view codes and two-dimensional image features corresponding to the cameras are processed by a single-carriage bird's-eye view converter to obtain current bird's-eye view features of the carriage;

[0007] The current bird's-eye view features of the multiple carriages are fused to obtain multi-carriage fusion features;

[0008] Obtaining obstacle information of the multi-carriage vehicle group using the multi-carriage fusion feature;

[0009] The processing includes:

[0010] Calculating the camera parameters and the first two-dimensional image features to obtain a first key vector and a first value vector;

[0011] Obtain a first to-be-processed bird's-eye view feature based on the bird's-eye view code, and obtain a second to-be-processed bird's-eye view feature based on the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature;

[0012] The camera parameters and the second two-dimensional image features are calculated to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0013] Processing the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature;

[0014] The second intermediate feature is processed to obtain a current bird's-eye view feature of the carriage.

[0015] Optionally, the single-car bird's-eye view converter includes a first fused attention module, a first convolution module, a first fully connected network module, a second fully connected network module, a second fused attention module, and a second convolution module;

[0016] Calculating the camera parameters and the first two-dimensional image features using the first fully connected network module to obtain a first key vector and a first value vector;

[0017] Using the first fusion attention module to process the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature to obtain a first intermediate feature;

[0018] Using the first convolution module to process the first intermediate feature to obtain a second bird's-eye view feature to be processed;

[0019] The camera parameters and the second two-dimensional image features are calculated using the second fully connected network module to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0020] Using the second fusion attention module to process the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature;

[0021] The second intermediate feature is processed by the second convolution module to obtain a current bird's-eye view feature of the carriage.

[0022] Optionally, the first fusion attention module and the second fusion attention module both include a first deformable attention unit and a second deformable attention unit;

[0023] Also includes:

[0024] Divide the first to-be-processed bird's-eye view features into grids of size P×P, and extract features in each of the grids as inputs of the first deformable attention unit;

[0025] Divide the P×P grid into an average number of G×G grids, and extract features from multiple associated grids as inputs of the second deformable attention unit;

[0026] P and G are both integers greater than 1.

[0027] Optionally, the process of fusing the current bird's-eye view features of multiple carriages to obtain multi-carriage fusion features includes:

[0028] Obtaining the angle of the hinge chain between every two adjacent carriages;

[0029] Obtaining the splicing features of the multiple carriages based on all the hinge chain angles and all the current bird's-eye view features;

[0030] The splicing features are fused to obtain multi-carriage fusion features.

[0031] Optionally, the process of obtaining the splicing features of the multiple carriages based on all the hinge link angles and all the current bird's-eye view features includes:

[0032] According to the angle of the hinge chain between the i-th carriage and the i-1-th carriage, the current bird's-eye view features of the i-th carriage are converted to the bird's-eye view space coordinate system of the front carriage to obtain the features to be spliced ​​of the i-th carriage; i=2, 3…, n, where n is the total number of carriages in the multi-unit vehicle;

[0033] The current bird's-eye view feature of the first carriage is determined as the to-be-joined feature of the first carriage;

[0034] The features to be spliced ​​of the multiple carriages are spliced ​​to obtain splicing features.

[0035] Optionally, the process of fusing the splicing features to obtain multi-carriage fusion features includes:

[0036] The splicing features are input into the bird's-eye view fusion converter to obtain multi-carriage fusion features.

[0037] Optionally, the process of obtaining the obstacle information of the multi-carriage vehicle group by using the multi-carriage fusion feature includes:

[0038] The multi-carriage fusion feature is input into a bird's-eye view decoder to obtain obstacle information of the multi-carriage vehicle, wherein the obstacle information includes the three-dimensional coordinates of the obstacle in the whole vehicle coordinate system of the multi-carriage vehicle, the length, width and height of the obstacle, and the heading angle.

[0039] Optionally, the obstacle information also includes the category of the obstacle.

[0040] Optionally, also include:

[0041] Automatic driving control is performed on the multi-group vehicle based on the obstacle information.

[0042] On the other hand, the present invention further provides an obstacle recognition system for a multi-unit vehicle, wherein the multi-unit vehicle comprises a plurality of articulated carriages, each of which is provided with a plurality of cameras, the cameras being used to collect sub-surround view images within their own collection areas, and the obstacle recognition system for the multi-unit vehicle comprises:

[0043] a processing module, configured to obtain, for each of the carriages, the sub-surround view images collected by the multiple cameras on the carriage, extract two-dimensional image features from all the sub-surround view images, and process the camera parameters, the bird's-eye view code and the two-dimensional image features corresponding to the cameras through a single-carriage bird's-eye view converter to obtain current bird's-eye view features of the carriage;

[0044] A fusion module is used to fuse the current bird's-eye view features of multiple carriages to obtain multi-carriage fusion features;

[0045] A decoding module, used for obtaining obstacle information of the multi-carriage vehicle group by using the multi-carriage fusion feature;

[0046] The processing includes:

[0047] Calculating the camera parameters and the first two-dimensional image features to obtain a first key vector and a first value vector;

[0048] Obtain a first to-be-processed bird's-eye view feature based on the bird's-eye view code, and obtain a second to-be-processed bird's-eye view feature based on the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature;

[0049] The camera parameters and the second two-dimensional image features are calculated to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0050] Processing the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature;

[0051] The second intermediate feature is processed to obtain a current bird's-eye view feature of the carriage.

[0052] On the other hand, the present invention further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the obstacle identification method for multiple vehicle groups as described in any one of the above items.

[0053] In another aspect, the present invention further provides an electronic device, comprising:

[0054] Memory for storing computer programs;

[0055] A processor is used to implement the steps of the obstacle recognition method for multiple vehicle groups as described in any one of the above items when executing the computer program.

[0056] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the obstacle recognition method for multi-unit vehicles as described in any one of the above items are implemented.

[0057] The present invention provides an obstacle recognition method for a multi-unit vehicle, which realizes three-dimensional perception of obstacles around a multi-unit articulated vehicle by a camera arranged on the multi-unit vehicle. Compared with a laser radar, the camera has a lower cost and a longer life. The current bird's-eye view features of the vehicle are obtained based on the sub-surround view images collected by the camera of each vehicle, and the current bird's-eye view features of multiple vehicles are fused to obtain multi-vehicle fusion features. The non-overlapping fields of view from different cameras are fused into a complete scene representation to fill the blind spots of a single camera, which is convenient for subsequent global understanding and analysis, so that the obstacle information around the vehicle can be accurately identified, and the reliability and comprehensiveness of obstacle perception can be improved. The present invention also provides an obstacle recognition system, electronic equipment, computer program product and computer-readable storage medium for a multi-unit vehicle, which have the same beneficial effects as the obstacle recognition method for a multi-unit vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A flowchart of the steps of an obstacle identification method for a multi-train group provided by the present invention;

[0060] Figure 2 A schematic diagram of camera layout provided by the present invention;

[0061] Figure 3A schematic diagram of a single-carriage bird's-eye view converter provided by the present invention;

[0062] Figure 4 A schematic diagram of a fusion attention module provided by the present invention;

[0063] Figure 5 A schematic diagram of local extraction of the first deformable attention unit provided by the present invention;

[0064] Figure 6 A schematic diagram of global extraction of the second deformable attention unit provided by the present invention;

[0065] Figure 7 A schematic diagram of a bird's-eye view fusion converter provided by the present invention;

[0066] Figure 8 A schematic diagram of an obstacle recognition process for a multi-train vehicle provided by the present invention;

[0067] Fig. 9 A schematic diagram of the structure of an obstacle recognition system for multiple trains provided by the present invention;

[0068] Fig.10 A schematic diagram of the structure of an electronic device provided by the present invention;

[0069] Fig.11 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0070] The core of the present invention is to provide an obstacle recognition method, system, equipment, product and medium for multi-train vehicles, which can reduce the obstacle perception cost and improve the reliability and comprehensiveness of obstacle perception.

[0071] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] First, please refer to Figure 1 The present invention provides an obstacle recognition method for a multi-train vehicle. The multi-train vehicle includes a plurality of articulated carriages. Each carriage is provided with a plurality of cameras. The cameras are used to collect sub-surround view images within their own collection areas. The obstacle recognition method for the multi-train vehicle includes:

[0073] S101: for each carriage, sub-surround view images collected by multiple cameras on the carriage are obtained, two-dimensional image features are extracted from all sub-surround view images, and camera parameters, bird's-eye view codes and two-dimensional image features corresponding to the cameras are processed by a single-carriage bird's-eye view converter to obtain current bird's-eye view features of the carriage;

[0074] Processing includes:

[0075] Calculating the camera parameters and the first two-dimensional image features to obtain a first key vector and a first value vector;

[0076] Obtain a first to-be-processed bird's-eye view feature based on the bird's-eye view encoding, and obtain a second to-be-processed bird's-eye view feature based on the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature;

[0077] The camera parameters and the second two-dimensional image features are calculated to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0078] Processing the second key vector, the second value vector and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature;

[0079] The second intermediate feature is processed to obtain the current bird's-eye view feature of the carriage.

[0080] In this embodiment, the multi-unit vehicle may be a multi-unit articulated vehicle, that is, a vehicle formed by multiple carriages articulated by an articulation device. In order to reduce the cost of obstacle perception, this embodiment uses multiple cameras arranged on the multi-unit vehicle to collect sub-surround view images around the vehicle body, so that a complete surround view image of the multi-unit vehicle can be obtained based on the sub-surround view images collected by the multiple cameras. As an optional embodiment, at least one camera may be arranged on the left and right sides of the carriage to collect the sub-surround view images of the side of the carriage, a camera may be arranged on the front side of the front carriage to collect the sub-surround view images from the front view perspective, and a camera may be arranged on the rear side of the rear carriage to collect the sub-surround view images from the rear view perspective. Figure 2 As shown, the multi-unit articulated vehicle includes three carriages, namely the front carriage, the rear carriage and the middle carriage. The front carriage and the middle carriage are connected by a hinge, and the rear carriage and the middle carriage are connected by a hinge. Five cameras are arranged in the front carriage and the rear carriage respectively, and two cameras are arranged on the left side and the right side of the middle carriage respectively to fill the blind spots of a single camera.

[0081] Among them, each camera can collect the sub-surround view image of its corresponding collection area according to the preset sampling frequency. The sampling frequency can be set according to the actual project needs, and this embodiment does not make any specific limitations here.

[0082] In order to make it easier to accurately describe the position and size of obstacles in space, and to make the spatial relationship between obstacles and multi-group vehicles more intuitive, this embodiment converts the image captured by the camera from the original view (such as front view, side view) into a bird's-eye view feature to improve the accuracy of obstacle detection, facilitate the subsequent more accurate positioning of the obstacle, and thus facilitate path planning and obstacle avoidance in the automatic driving system. It can be understood that each car is equipped with multiple cameras, and multiple cameras can capture sub-surround view images at the same time. The sub-surround view images captured by multiple cameras are extracted with two-dimensional image features. Feature extraction helps to remove noise and irrelevant information in the original sub-surround view images, better align features under different perspectives, ensure the consistency of spatial information, and thus improve the accuracy of obstacle detection. In addition, considering that directly performing BEV (Bird's Eye View) conversion on the original image may involve a large number of geometric transformations and image processing operations, extracting two-dimensional image features first and performing BEV conversion on the two-dimensional image features can reduce the amount of calculation.

[0083] Specifically, in this embodiment, the sub-surround view images collected by multiple cameras of a single compartment can be subjected to feature extraction by a single compartment feature extractor. The input of the single compartment feature extractor is the sub-surround view images collected by each camera of the single compartment, which is assumed to be N×H×W×C, where N is the number of cameras on the compartment, H is the height of the sub-surround view image, W is the width of the sub-surround view image, and C is the channel of the sub-surround view image. The output of the single compartment feature extractor is a two-dimensional image feature, and its feature size is N×H'×W'×C', where N is the number of cameras on the compartment, H' is the height of the two-dimensional image feature, W' is the width of the two-dimensional image feature, and C' is the channel of the two-dimensional image feature. ResNet (Residual Network), EfficientNet (a convolutional neural network), VoVNet (Volumetric Overlapped VotingNetwork), etc. can be used mainly to extract the two-dimensional image features of the sub-surround view image.

[0084] In an exemplary embodiment, the process of converting the two-dimensional image features into the current bird's-eye view features of the vehicle cabin includes:

[0085] Get the camera parameters corresponding to the camera;

[0086] The 2D image features are converted into the current bird's-eye view features of the vehicle cabin based on the camera parameters.

[0087] The camera parameters in this embodiment include the intrinsic parameters and extrinsic parameters of the camera. The intrinsic parameters of the camera describe the inherent properties of the camera itself and are independent of the external environment of the camera. These parameters include but are not limited to focal length, principal point coordinates, radial distortion coefficient, tangential distortion coefficient, image scale factor, etc. The extrinsic parameters of the camera describe the position and direction of the camera relative to a global coordinate system. These parameters include but are not limited to rotation parameters, translation parameters, etc. The camera parameters are used to project the two-dimensional image features to the bird's-eye view perspective, ensuring the accurate mapping from the two-dimensional image features to the current BEV features, so that the features maintain the correct spatial relationship and scale during the conversion process. In addition, in a multi-camera perception system, as an optional embodiment, the intrinsic and extrinsic parameters of each camera need to be accurately calibrated so that the image features captured by different cameras can be fused into a unified BEV representation.

[0088] In an exemplary embodiment, the process of converting the two-dimensional image features into the current bird's-eye view features of the vehicle cabin based on the camera parameters includes:

[0089] The camera parameters, bird's-eye view code and two-dimensional image features are input into the single-carriage bird's-eye view converter corresponding to the car, and the current bird's-eye view features of the car are obtained.

[0090] In this embodiment, an independent single-carriage bird's-eye view converter is configured for each car to realize the conversion of two-dimensional image features to current bird's-eye view features, thereby improving the conversion efficiency.

[0091] Among them, the bird's-eye view encoding represents the features from the perspective of a single vehicle bird's-eye view (BEV), which is a learnable vector.

[0092] In an exemplary embodiment, referring to Figure 3 As shown, the single-car bird's-eye view converter includes a first fused attention module, a first convolution module, a first fully connected network module, a second fully connected network module, a second fused attention module, and a second convolution module;

[0093] The process of inputting the camera parameters, the bird's-eye view code and the two-dimensional image features into the single-carriage bird's-eye view converter corresponding to the car, and obtaining the current bird's-eye view features of the car includes:

[0094] Calculate the camera parameters and the first two-dimensional image features using the first fully connected network module to obtain a first key vector and a first value vector;

[0095] Obtaining a first to-be-processed bird's-eye view feature based on the bird's-eye view coding;

[0096] Using a first fusion attention module, the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature are processed to obtain a first intermediate feature;

[0097] Using the first convolution module to process the first intermediate feature to obtain a second bird's-eye view feature to be processed;

[0098] The camera parameters and the second two-dimensional image features are calculated using a second fully connected network module to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0099] Using a second fusion attention module, the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature are processed to obtain a second intermediate feature;

[0100] The second intermediate feature is processed by the second convolution module to obtain the current bird's-eye view feature of the car.

[0101] In this embodiment, the structure of the single-car bird's-eye view converter is as follows: Figure 3 As shown in the figure, its input is the bird's-eye view encoding, camera parameters, and two-dimensional image features, and its output is the current bird's-eye view features of a single car. Its purpose is to convert the two-dimensional image features to the BEV perspective. Figure 3 In the figure, the bird's-eye view encoding is used to calculate the correlation with the image features. The key vector K and the value vector V are calculated by the two-dimensional image features and the camera internal and external parameters through a fully connected network. The key vector K and the value vector V are calculated by the bird's-eye view encoding to calculate the correlation, and the three are input into the fusion attention module. Figure 3 In the figure, downsampling by 4 times represents the feature downsampling rate corresponding to the first two-dimensional image feature, downsampling by 8 times represents the feature downsampling rate corresponding to the second two-dimensional image feature, Q1, K1, and V1 represent the first bird's-eye view feature to be processed, the first key vector, and the first value vector respectively, and Q2, K2, and V2 represent the second bird's-eye view feature to be processed, the second key vector, and the second value vector respectively.

[0102] S102: Fusing the current bird's-eye view features of multiple carriages to obtain multi-carriage fusion features;

[0103] In this embodiment, after obtaining the current bird's-eye view features of a single car, the current bird's-eye view features of multiple cars are fused to provide more comprehensive coverage around the vehicle, reduce blind spots, and enhance environmental perception capabilities. Considering that cameras in different cars may have their own advantages and limitations due to different positions and viewing angles, fusing the current bird's-eye view features of each car can complement these differences. The fused multi-carriage fusion features are represented in a unified coordinate system, which improves the robustness of perception and more accurately detects surrounding obstacles. At the same time, it helps to achieve coordinated control between cars and improve driving stability and efficiency.

[0104] S103: Obtain obstacle information of multiple vehicle groups using multi-carriage fusion features.

[0105] In this embodiment, the multi-carriage fusion feature can be used to perform a global understanding and analysis to obtain the obstacle information of the multi-carriage vehicle. The obstacle information includes but is not limited to the three-dimensional coordinates, length, width, height and heading angle of the obstacle in the vehicle coordinate system. The obstacle information is then sent through the local area network within the multi-carriage vehicle so that the vehicle control unit in the multi-carriage vehicle can avoid obstacles according to the obstacle information and realize automatic driving.

[0106] It can be seen that in this embodiment, three-dimensional perception of obstacles around multi-carriage articulated vehicles is achieved by cameras installed on multi-carriage vehicles. Compared with lidar, cameras have lower cost and longer life. The current bird's-eye view features of each car are obtained based on the sub-surround view images collected by the camera of each car, and the current bird's-eye view features of multiple cars are fused to obtain multi-carriage fusion features. The non-overlapping fields of view from different cameras are fused into a complete scene representation to fill the blind spots of a single camera, which is convenient for subsequent global understanding and analysis, so that the obstacle information around the vehicle can be accurately identified, and the reliability and comprehensiveness of obstacle perception can be improved.

[0107] Based on the above embodiments:

[0108] In an exemplary embodiment, the first fused attention module and the second fused attention module each include a first deformable attention unit and a second deformable attention unit;

[0109] Also includes:

[0110] Divide the first bird's-eye view feature to be processed into a grid of size P×P, and extract the features in each grid as the input of the first deformable attention unit;

[0111] The P×P grid is divided into an average of G×G grids, and the features in multiple associated grids are extracted as the input of the second deformable attention unit. P and G are both integers greater than 1.

[0112] Reference Figure 4 , Figure 4 Shows the l-1th layer vector z of the fused attention module (including the first fused attention module and the second fused attention module) l-1 To the l+1 layer vector z l+1 The calculation process of the fusion attention module is composed of layer normalization (LayerNorm), deformable attention unit, and multilayer perceptron (MLP). The first deformable attention unit and the second deformable attention unit focus on the local and global bird's-eye view features respectively. The specific extraction position is referred to Figure 5 and Figure 6 shown.

[0113] Reference Figure 5 As shown in the figure, the first deformable attention unit local extraction divides the first to-be-processed bird's-eye view features into P×P fixed-size grids, and extracts the features in each grid as the input of the first deformable attention unit. This can effectively focus on the features of different cameras in a single compartment at the same grid position, which is helpful to solve the occlusion problem of a single camera.

[0114] Reference Figure 6 As shown in , the second deformable attention unit globally extracts the P×P fixed grid into an average G×G number of grids, and extracts features from multiple associated grids as the input of the second deformable attention unit, as shown in Figure 6 As shown in the figure, p11, p21, p31, p41, p51, p61, p71, and p81 are a group of associated grids, and the rest of the groups of associated networks are similar. This allows the neural network to focus on the global information of the current scene, reducing the amount of calculation, which is beneficial for the neural network to understand the position distribution of each camera in a single car and better integrate the information of multiple cameras.

[0115] Q, K, and V are calculated as follows:

[0116] Deformable Attention Unit (Q, K, V) = ;

[0117] Where Q is the feature of the bird's-eye view to be processed, K is the key vector, V is the value vector, softmax is the normalized exponential function, and d k is the vector dimension.

[0118] In an exemplary embodiment, the process of fusing the current bird's-eye view features of multiple carriages to obtain the multi-carriage fusion features includes:

[0119] Get the angle of the hinge chain between every two adjacent carriages;

[0120] Get the splicing features of multiple carriages based on all articulated chain angles and all current bird's-eye view features;

[0121] The splicing features are fused to obtain multi-carriage fusion features;

[0122] The process of obtaining the splicing features of multiple carriages based on all articulated chain angles and all current bird's-eye view features includes:

[0123] According to the angle of the hinge chain between the i-th carriage and the i-1-th carriage, the current bird's-eye view features of the i-th carriage are converted to the bird's-eye view space coordinate system of the front carriage to obtain the features to be spliced ​​of the i-th carriage; i=2, 3…, n, where n is the total number of carriages in the multi-unit vehicle;

[0124] The current bird's-eye view feature of the first carriage is determined as the to-be-joined feature of the first carriage;

[0125] The features to be spliced ​​of multiple carriages are spliced ​​together to obtain splicing features.

[0126] In this embodiment, the current bird's-eye view features of the current car are converted to the bird's-eye view space coordinate system of the front car according to the hinge chain angle between the current car and the previous car. This can eliminate the coordinate deviation caused by the position and direction differences between different cars, ensure that the sub-surround view information around the entire vehicle is in the same reference system, and splice the converted features to be spliced ​​to obtain spliced ​​features. The spliced ​​bird's-eye view features can provide a complete panoramic view around the vehicle, thereby enhancing the vehicle's perception of the surrounding environment, reducing blind spots, and improving driving safety.

[0127] It can be understood that using the front compartment of the vehicle as a reference point can more intuitively assist the autonomous driving system in making decisions such as steering, acceleration and braking.

[0128] In an exemplary embodiment, the process of fusing the splicing features to obtain the multi-carriage fusion features includes:

[0129] The spliced ​​features are input into the bird’s-eye view fusion transformer to obtain multi-carriage fusion features.

[0130] Structural reference of Bird's Eye View Fusion Converter Figure 7 As shown in the figure, the bird's-eye view fusion converter is similar to the single-car bird's-eye view converter, and is also composed of layer normalization, deformable attention units, and multi-layer perceptrons. The difference is that its input is concatenated features, and the query vector, key vector, and value vector input to the local attention unit and the global attention unit are all obtained from the BEV features through different fully connected layers. Since the field of view of each car is different, paying attention to the local area of ​​the same BEV can make up for the blind spots that may exist in a single car. In addition, the global attention unit with sparse sampling can interact globally to obtain contextual understanding of the map semantics, and finally obtain multi-car fusion features.

[0131] In an exemplary embodiment, the process of obtaining obstacle information of multiple vehicle groups by using multi-carriage fusion features includes:

[0132] The multi-carriage fusion features are input into the bird's-eye view decoder to obtain the obstacle information of the multi-carriage vehicle. The obstacle information includes the three-dimensional coordinates of the obstacle in the whole vehicle coordinate system of the multi-carriage vehicle, the length, width and height of the obstacle, and the heading angle.

[0133] In an exemplary embodiment, the obstacle information further includes the category of the obstacle.

[0134] After obtaining the multi-carriage fusion features, the multi-carriage fusion features are sent to the BEV decoder to obtain the obstacle information around the multi-carriage vehicle. The BEV decoder consists of two 1x1 convolutional layers, which perform box regression and classification respectively. The regression output is (x, y, z, w, d, h, θ), which represent the 3D position, size (length, width and height) and yaw angle respectively. The classification output is the confidence score of each anchor box as an object or background, which is used to distinguish the category of obstacles. Finally, the obtained obstacle information is sent to the vehicle control unit of the multi-carriage vehicle to help the multi-carriage vehicle avoid obstacles and realize automatic driving. Reference Figure 8 As shown, Figure 8 Is for Figure 2 The diagram shows the obstacle recognition process flow of a multi-unit vehicle.

[0135] In summary, the present invention provides a set of surround perception solutions for multi-unit articulated vehicles, and realizes 3D perception of obstacles around multi-unit articulated vehicles through a surround view camera system including multiple cameras. The low-cost camera solution is used to perceive obstacles, which can effectively reduce costs compared to the lidar solution. In response to the need for 3D obstacle perception for autonomous driving of multi-unit articulated vehicles, a 3D obstacle perception solution based on a transformer (a model architecture based on a self-attention mechanism) network structure is provided. The images collected by the cameras of each car are used as input, and the 3D information of obstacles around the vehicle is output to provide guidance for the planning and control module of the autonomous driving, thereby ensuring the safe driving of multi-unit articulated trains. In response to the demand for multi-carriage perception fusion of multi-carriages in multi-articulated trains, a multi-carriage fusion perception solution was proposed. By focusing on the overall surround features of the multi-carriage articulated train and the surround features of a single car, the perception of obstacles around the entire vehicle body is completed. This perception solution can effectively integrate the perception range of multiple cameras and multiple cars to fill the blind spots of a single camera, and can combine the global position information of multiple cameras to better perceive the obstacle information around multi-articulated vehicles.

[0136] Second, please refer to Fig. 9 The present invention also provides an obstacle recognition system for a multi-unit vehicle, wherein the multi-unit vehicle comprises a plurality of articulated carriages, each carriage is provided with a plurality of cameras, and the cameras are used to collect sub-surround view images within their own collection areas. The obstacle recognition system for the multi-unit vehicle comprises:

[0137] The processing module 11 is used to obtain sub-surround view images collected by multiple cameras on each carriage, extract two-dimensional image features from all sub-surround view images, and process the camera parameters, bird's-eye view codes and two-dimensional image features corresponding to the cameras through a single-carriage bird's-eye view converter to obtain the current bird's-eye view features of the carriage;

[0138] A fusion module 12 is used to fuse the current bird's-eye view features of multiple carriages to obtain multi-carriage fusion features;

[0139] A decoding module 13, used to obtain obstacle information of multiple train groups by using multi-carriage fusion features;

[0140] The processing includes:

[0141] Calculating the camera parameters and the first two-dimensional image features to obtain a first key vector and a first value vector;

[0142] Obtain a first to-be-processed bird's-eye view feature based on the bird's-eye view code, and obtain a second to-be-processed bird's-eye view feature based on the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature;

[0143] The camera parameters and the second two-dimensional image features are calculated to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0144] Processing the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature;

[0145] The second intermediate feature is processed to obtain a current bird's-eye view feature of the carriage.

[0146] In an exemplary embodiment, the single-car bird's-eye view converter includes a first fused attention module, a first convolution module, a first fully connected network module, a second fully connected network module, a second fused attention module, and a second convolution module;

[0147] The process of inputting the camera parameters, the bird's-eye view code and the two-dimensional image features into the single-carriage bird's-eye view converter corresponding to the car, and obtaining the current bird's-eye view features of the car includes:

[0148] Calculate the camera parameters and the first two-dimensional image features using the first fully connected network module to obtain a first key vector and a first value vector;

[0149] Obtaining a first to-be-processed bird's-eye view feature based on the bird's-eye view coding;

[0150] Using a first fusion attention module, the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature are processed to obtain a first intermediate feature;

[0151] Using the first convolution module to process the first intermediate feature to obtain a second bird's-eye view feature to be processed;

[0152] The camera parameters and the second two-dimensional image features are calculated using a second fully connected network module to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features;

[0153] Using a second fusion attention module, the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature are processed to obtain a second intermediate feature;

[0154] The second intermediate feature is processed by the second convolution module to obtain the current bird's-eye view feature of the car.

[0155] In an exemplary embodiment, the first fused attention module and the second fused attention module each include a first deformable attention unit and a second deformable attention unit;

[0156] Also includes:

[0157] A first partitioning module is used to partition the first to-be-processed bird's-eye view feature into grids of size P×P, and extract features in each grid as input of the first deformable attention unit;

[0158] The second division module is used to divide the P×P grid into an average of G×G grids, extract features from multiple associated grids as inputs of the second deformable attention unit, and both P and G are integers greater than 1.

[0159] In an exemplary embodiment, the process of fusing the current bird's-eye view features of multiple carriages to obtain the multi-carriage fusion features includes:

[0160] Get the angle of the hinge chain between every two adjacent carriages;

[0161] Get the splicing features of multiple carriages based on all articulated chain angles and all current bird's-eye view features;

[0162] The splicing features are fused to obtain multi-carriage fusion features.

[0163] In an exemplary embodiment, the process of obtaining the splicing features of multiple carriages based on all articulation link angles and all current bird's-eye view features includes:

[0164] According to the angle of the hinge chain between the i-th carriage and the i-1-th carriage, the current bird's-eye view features of the i-th carriage are converted to the bird's-eye view space coordinate system of the front carriage to obtain the features to be spliced ​​of the i-th carriage; i=2, 3…, n, where n is the total number of carriages in the multi-unit vehicle;

[0165] The current bird's-eye view feature of the first carriage is determined as the to-be-joined feature of the first carriage;

[0166] The features to be spliced ​​of multiple carriages are spliced ​​together to obtain splicing features.

[0167] In an exemplary embodiment, the process of fusing the splicing features to obtain the multi-carriage fusion features includes:

[0168] The spliced ​​features are input into the bird’s-eye view fusion transformer to obtain multi-carriage fusion features.

[0169] In an exemplary embodiment, the process of obtaining obstacle information of multiple vehicle groups by using multi-carriage fusion features includes:

[0170] The multi-carriage fusion features are input into the bird's-eye view decoder to obtain the obstacle information of the multi-carriage vehicle. The obstacle information includes the three-dimensional coordinates of the obstacle in the whole vehicle coordinate system of the multi-carriage vehicle, the length, width and height of the obstacle, and the heading angle.

[0171] In an exemplary embodiment, the obstacle information further includes the category of the obstacle.

[0172] In an exemplary embodiment, it further includes:

[0173] A control module is used to perform automatic driving control of multiple vehicles based on obstacle information.

[0174] In a third aspect, the present invention further provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the obstacle recognition method for multiple vehicle groups as described in any of the embodiments above.

[0175] For an introduction to a computer program product provided by the present invention, please refer to the above embodiments, and the present invention will not be described in detail here.

[0176] A computer program product provided by the present invention has the same beneficial effects as the obstacle recognition method for multiple vehicle groups mentioned above.

[0177] Fourth, please refer to Fig.10 The present invention also provides an electronic device, comprising:

[0178] A memory 21, used for storing computer programs;

[0179] The processor 22 is used to implement the steps of the obstacle recognition method for multiple vehicle groups as described in any of the above embodiments when executing the computer program.

[0180] The electronic device also includes:

[0181] The input interface 23 is connected to the processor 22 via the communication bus 26, and is used to obtain the computer programs, parameters and instructions imported from the outside, and save them in the memory 21 under the control of the processor 22. The input interface can be connected to an input device to receive parameters or instructions manually input by the user. The input device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the terminal housing.

[0182] The display unit 24 is connected to the processor 22 via the communication bus 26 and is used to display the data sent by the processor 22. The display unit can be a liquid crystal display or an electronic ink display.

[0183] The network port 25 is connected to the processor 22 via the communication bus 26, and is used to communicate with various external terminal devices. The communication technology used in the communication connection can be a wired communication technology or a wireless communication technology, such as mobile high-definition link technology, universal serial bus, high-definition multimedia interface, wireless fidelity technology, Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s, etc.

[0184] For an introduction to an electronic device provided by the present invention, please refer to the above embodiments, and the present invention will not be described in detail here.

[0185] The electronic device provided by the present invention has the same beneficial effects as the obstacle recognition method for multiple-unit vehicles mentioned above.

[0186] Fifth, please refer to Fig.11 The present invention further provides a computer-readable storage medium 30, on which a computer program 31 is stored. When the computer program 31 is executed by a processor, the steps of the obstacle recognition method for multiple vehicles as described in any of the above embodiments are implemented.

[0187] The computer-readable storage medium 30 may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0188] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above embodiments, and the present invention will not be described in detail here.

[0189] The computer-readable storage medium provided by the present invention has the same beneficial effects as the obstacle recognition method for multiple vehicle groups mentioned above.

[0190] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0191] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying obstacles in a multi-unit vehicle, wherein the multi-unit vehicle comprises a plurality of articulated carriages, characterized in that: Each carriage is provided with a plurality of cameras, and the cameras are used to collect sub-surround view images within their own collection areas. The complete surround view image of the multi-unit vehicle is obtained based on the sub-surround view images collected by the plurality of cameras. The obstacle recognition method of the multi-unit vehicle includes: For each of the carriages, the sub-surround view images collected by the multiple cameras on the carriage are obtained, two-dimensional image features are extracted from all the sub-surround view images, and the camera parameters, bird's-eye view codes and two-dimensional image features corresponding to the cameras are processed by a single-carriage bird's-eye view converter to obtain current bird's-eye view features of the carriage; The current bird's-eye view features of the multiple carriages are fused to obtain multi-carriage fusion features; Obtaining obstacle information of the multi-carriage vehicle group using the multi-carriage fusion feature; The processing includes: Calculating the camera parameters and the first two-dimensional image features to obtain a first key vector and a first value vector; Obtain a first to-be-processed bird's-eye view feature based on the bird's-eye view code, and obtain a second to-be-processed bird's-eye view feature based on the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature; The camera parameters and the second two-dimensional image features are calculated to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features; Processing the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature; Processing the second intermediate feature to obtain a current bird's-eye view feature of the carriage; The process of fusing the current bird's-eye view features of multiple carriages to obtain the multi-carriage fusion features includes: Obtaining the angle of the hinge chain between every two adjacent carriages; According to the angle of the hinge chain between the i-th carriage and the i-1-th carriage, the current bird's-eye view features of the i-th carriage are converted to the bird's-eye view space coordinate system of the front carriage to obtain the features to be spliced ​​of the i-th carriage; i=2, 3…, n, where n is the total number of carriages in the multi-unit vehicle; The current bird's-eye view feature of the first carriage is determined as the to-be-joined feature of the first carriage; Splicing the features to be spliced ​​of the multiple carriages to obtain splicing features; The splicing features are fused to obtain multi-carriage fusion features; The process of obtaining the obstacle information of the multi-carriage vehicle group by using the multi-carriage fusion feature includes: The multi-carriage fusion features are input into a bird's-eye view decoder to obtain obstacle information of the multi-carriage vehicle group.

2. The obstacle recognition method for multiple vehicles according to claim 1, characterized in that: The single-carriage bird's-eye view converter includes a first fused attention module, a first convolution module, a first fully connected network module, a second fully connected network module, a second fused attention module and a second convolution module; Calculating the camera parameters and the first two-dimensional image features using the first fully connected network module to obtain a first key vector and a first value vector; Using the first fusion attention module to process the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature to obtain a first intermediate feature; Using the first convolution module to process the first intermediate feature to obtain a second bird's-eye view feature to be processed; The camera parameters and the second two-dimensional image features are calculated using the second fully connected network module to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features; Using the second fusion attention module to process the second key vector, the second value vector and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature; The second intermediate feature is processed by the second convolution module to obtain a current bird's-eye view feature of the carriage.

3. The obstacle recognition method for multiple trains according to claim 2, characterized in that: The first fused attention module and the second fused attention module each include a first deformable attention unit and a second deformable attention unit; Also includes: Divide the first to-be-processed bird's-eye view features into grids of size P×P, and extract features in each of the grids as inputs of the first deformable attention unit; Divide the P×P grid into an average number of G×G grids, and extract features from multiple associated grids as inputs of the second deformable attention unit; P and G are both integers greater than 1.

4. The obstacle recognition method for multiple vehicles according to claim 1, characterized in that: The process of fusing the splicing features to obtain multi-carriage fusion features includes: The splicing features are input into the bird's-eye view fusion converter to obtain multi-carriage fusion features.

5. The obstacle recognition method for multiple vehicles according to claim 1, characterized in that: The obstacle information includes the three-dimensional coordinates of the obstacle in the whole vehicle coordinate system of the multi-unit vehicle, the length, width and height of the obstacle, and the heading angle.

6. The obstacle recognition method for multiple train sets according to claim 5, characterized in that: The obstacle information also includes the category of the obstacle.

7. The obstacle recognition method for a multi-unit vehicle according to any one of claims 1 to 6, characterized in that: Also includes: Automatic driving control is performed on the multi-group vehicle based on the obstacle information.

8. An obstacle recognition system for a multi-unit train, the multi-unit train comprising a plurality of articulated carriages, characterized in that: Each carriage is provided with a plurality of cameras, and the cameras are used to collect sub-surround view images within their own collection areas. The complete surround view image of the multi-unit vehicle is obtained based on the sub-surround view images collected by the plurality of cameras. The obstacle recognition system of the multi-unit vehicle includes: a processing module, configured to obtain, for each of the carriages, the sub-surround view images collected by the multiple cameras on the carriage, extract two-dimensional image features from all the sub-surround view images, and process the camera parameters, the bird's-eye view code and the two-dimensional image features corresponding to the cameras through a single-carriage bird's-eye view converter to obtain current bird's-eye view features of the carriage; A fusion module is used to fuse the current bird's-eye view features of multiple carriages to obtain multi-carriage fusion features; A decoding module, used for obtaining obstacle information of the multi-carriage vehicle group by using the multi-carriage fusion feature; The processing includes: Calculating the camera parameters and the first two-dimensional image features to obtain a first key vector and a first value vector; Obtain a first to-be-processed bird's-eye view feature based on the bird's-eye view code, and obtain a second to-be-processed bird's-eye view feature based on the first key vector, the first value vector, and the first to-be-processed bird's-eye view feature; The camera parameters and the second two-dimensional image features are calculated to obtain a second key vector and a second value vector; the feature downsampling rate corresponding to the second two-dimensional image features is greater than the feature downsampling rate corresponding to the first two-dimensional image features; Processing the second key vector, the second value vector, and the second to-be-processed bird's-eye view feature to obtain a second intermediate feature; Processing the second intermediate feature to obtain a current bird's-eye view feature of the carriage; The process of fusing the current bird's-eye view features of multiple carriages to obtain the multi-carriage fusion features includes: Obtaining the angle of the hinge chain between every two adjacent carriages; According to the angle of the hinge chain between the i-th carriage and the i-1-th carriage, the current bird's-eye view features of the i-th carriage are converted to the bird's-eye view space coordinate system of the front carriage to obtain the features to be spliced ​​of the i-th carriage; i=2, 3…, n, where n is the total number of carriages in the multi-unit vehicle; The current bird's-eye view feature of the first carriage is determined as the to-be-joined feature of the first carriage; Splicing the features to be spliced ​​of the multiple carriages to obtain splicing features; The splicing features are fused to obtain multi-carriage fusion features; The process of obtaining the obstacle information of the multi-carriage vehicle group by using the multi-carriage fusion feature includes: The multi-carriage fusion features are input into a bird's-eye view decoder to obtain obstacle information of the multi-carriage vehicle group.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the obstacle recognition method for a multi-unit vehicle set as described in any one of claims 1-7 are implemented.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the obstacle recognition method for a multi-train as described in any one of claims 1-7 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the obstacle recognition method for a multi-unit vehicle set as described in any one of claims 1-7 are implemented.

Images