Vehicle environment detection method, device and equipment, vehicle, medium and product

Abstract

The invention relates to a vehicle environment detection method, device and equipment, a vehicle, a medium and a product, and the method comprises the steps: carrying out the feature extraction of a panoramic image and an all-round image of the vehicle, and obtaining a panoramic image feature and an all-round image feature; determining first query information based on a task type corresponding to the detection task; determining first key information and first value information based on the panoramic image feature and the panoramic image feature; processing the first query information, the first key information and the first value information by using an attention mechanism to obtain a target fusion feature; and detecting the surrounding environment of the vehicle based on the target fusion feature, and determining a detection result corresponding to the detection task. According to the method, the panoramic image features and the all-round image features are fused and utilized by using an attention mechanism to improve the perception accuracy, and the task related information, the panoramic image features and the all-round image features are subjected to sparse expression, so that the calculation amount and memory occupation in the feature fusion process are reduced.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a vehicle environment detection method, device, equipment, vehicle, medium and product. Background Art

[0002] When detecting the vehicle's surrounding environment, make full use of the image information from various perspectives on the vehicle to improve the comprehensive perception and positioning capabilities of the target and environment. Under the premise of supporting the driving and parking function requirements, the vehicle will be equipped with a panoramic camera and a surround view camera that can cover a 360° viewing angle around the vehicle to achieve detection coverage around the vehicle, especially at medium and short distances. In related technologies, due to the large differences between the panoramic image and the surround view image in terms of field of view, distortion mode, etc., the panoramic view and surround view are generally processed independently for driving and parking tasks, resulting in limited resource consumption on the vehicle side and high latency caused by preemption, as well as the inability to effectively compensate for the inherent blind spot characteristics of the panoramic view. Summary of the invention

[0003] The present application provides a vehicle environment detection method, device, equipment, vehicle, medium and product.

[0004] The technical solutions adopted in this application are as follows:

[0005] A vehicle environment detection method comprises: extracting features from a surrounding image and a surround image of the vehicle respectively to obtain a surrounding image feature and a surround image feature; determining first query information based on a task type corresponding to a detection task; determining first key information and first value information based on the surrounding image feature and the surround image feature; processing the first query information, the first key information and the first value information by using an attention mechanism to obtain a target fusion feature; and detecting the surrounding environment of the vehicle based on the target fusion feature to determine a detection result corresponding to the detection task.

[0006] According to the above technical means, firstly, the features of the vehicle's surrounding view image and surround view image are extracted respectively to obtain the surrounding view image features and surround view image features; secondly, the first query information is determined according to the task type corresponding to the detection task, and the first key information and the first value information corresponding to the detection task are determined according to the surrounding view image features and the surround view image features; the first query information, the first key information and the first value information are processed by using the attention mechanism to obtain the target fusion features; finally, according to the target fusion features, the surrounding environment of the vehicle is detected to determine the detection result corresponding to the detection task. In this way, on the one hand, the attention mechanism is used to integrate the input and output of the surrounding view image features and the surround view image features, which reduces the resource occupation and delayed waiting caused by independent operation, and effectively improves the information processing efficiency. Moreover, by constructing the key information and value information in the attention mechanism with the surrounding view image features and the surround view image features, the surrounding view image features and the surround view image features can be better integrated and utilized, thereby improving the perception ability in the process of environmental detection and improving the accuracy of environmental detection; on the other hand, the attention mechanism is used to sparsely express the task-related information, surrounding view image features, and surround view image features, thereby reducing the amount of computation and memory usage in the feature fusion process.

[0007] Furthermore, the surrounding view image features include two-dimensional surrounding view features and three-dimensional surrounding view position features, and the surround view image features include two-dimensional surround view features and three-dimensional surround view position features; feature extraction is performed on the surrounding view image and the surround view image of the vehicle respectively to obtain the surrounding view image features and the surround view image features, including: two-dimensional encoding the surrounding view image and the surround view image to determine the two-dimensional surrounding view features and the two-dimensional surround view features; three-dimensional encoding the surrounding view image and the surround view image to determine the three-dimensional surrounding view position features and the three-dimensional surround view position features.

[0008] According to the above technical means, the surrounding image features include two-dimensional surrounding features and three-dimensional surrounding position features, and the surround image features include two-dimensional surround features and three-dimensional surround position features; the surrounding image and the surround image are two-dimensionally encoded to determine the two-dimensional surrounding features and the two-dimensional surround features; the surrounding image and the surround image are three-dimensionally encoded to determine the three-dimensional surrounding position features and the three-dimensional surround position features. In this way, the surrounding image features and the surround image features can include complete two-dimensional image information and three-dimensional spatial position information, thereby improving the accuracy of environmental detection.

[0009] Further, based on the task type corresponding to the detection task, the first query information is determined, including: based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment, the second query information, the second key information and the second value information corresponding to the detection task are determined; and the second query information, the second key information and the second value information are processed by using the attention mechanism to obtain the first query information.

[0010] According to the above-mentioned technical means, the second query information, second key information and second value information corresponding to the detection task are determined according to the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment; the second query information, second key information and second value information are processed by using the attention mechanism to obtain the first query information, which can integrate the task-related information at the current moment and the task-related information at each historical moment.

[0011] Further, based on the task type corresponding to the detection task and the detection results corresponding to the detection tasks at each historical moment, the second query information, second key information and second value information corresponding to the detection task are determined, including: based on the posture conversion relationship between each historical moment and the vehicle at the current moment, the detection results corresponding to the detection tasks at each historical moment are reconstructed to determine the reconstruction features corresponding to the detection tasks at each historical moment; based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, the initial reconstruction features corresponding to the detection task at the current moment are determined; the first historical moment is the moment before the current moment; the initial reconstruction features corresponding to the detection task at the current moment are determined as the second query information; based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment, the second key information and second value information are determined; the second historical moment is a historical moment other than the first historical moment.

[0012] According to the above technical means, firstly, based on the posture conversion relationship between each historical moment and the vehicle at the current moment, the detection results corresponding to the detection tasks at each historical moment are reconstructed, the reconstruction features corresponding to the detection tasks at each historical moment are determined, and the initial reconstruction features corresponding to the detection tasks at the current moment are determined according to the task type corresponding to the detection task and the reconstruction features corresponding to the detection tasks at the first historical moment; the first historical moment is the previous moment of the current moment; secondly, the initial reconstruction features corresponding to the detection tasks at the current moment are determined as the second query information; finally, based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection tasks at the current moment, the second key information and the second value information are determined; the second historical moment is the historical moment other than the first historical moment. In this way, the second query information includes the initial reconstruction features corresponding to the detection tasks at the current moment, the second key information and the second value information include the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection tasks at the current moment, so that the task-related information at the current moment and the task-related information at each historical moment can be fused in the first query information, and then the association between the task-related information at the current moment and the task-related information at each historical moment can be established in the target fusion features, so as to further improve the accuracy of environmental detection.

[0013] Furthermore, based on the posture conversion relationship between the vehicle at each historical moment and the current moment, the detection results corresponding to the detection tasks at each historical moment are reconstructed, and the reconstruction features corresponding to the detection tasks at each historical moment are determined, including: based on the posture conversion relationship, the detection results corresponding to the detection tasks at each historical moment are aligned to the current moment, and the updated detection results corresponding to the detection tasks at each historical moment are determined; based on the posture conversion relationship, the time difference between each historical moment and the current moment, and the movement speed corresponding to the detection task, the updated detection results corresponding to the detection tasks at each historical moment are reconstructed, and the reconstruction features corresponding to the detection tasks at each historical moment are determined.

[0014] According to the above technical means, by aligning the detection results corresponding to the detection tasks at each historical moment to the current moment according to the posture conversion relationship, the updated detection results corresponding to the detection tasks at each historical moment are determined; according to the posture conversion relationship, the time difference between each historical moment and the current moment, and the movement speed corresponding to the detection task, the updated detection results corresponding to the detection tasks at each historical moment are reconstructed to determine the reconstructed features corresponding to the detection tasks at each historical moment. In this way, the detection results corresponding to the detection tasks at the historical moment are aligned to the current moment and reconstructed, so that the information related to the current moment task and the information related to the historical moment task maintain semantic continuity and consistency.

[0015] Furthermore, based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment, the initial reconstruction features corresponding to the detection task at the current moment are determined, including: based on the reconstruction features corresponding to the detection task at the first historical moment, determining the first initial reconstruction features corresponding to the detection task at the current moment; based on the task type corresponding to the detection task, randomly generating the second initial reconstruction features corresponding to the detection task at the current moment; and concatenating the first initial reconstruction features and the second initial reconstruction features to determine the initial reconstruction features corresponding to the current moment.

[0016] According to the above technical means, firstly, the first initial reconstruction feature corresponding to the detection task at the current moment is determined according to the reconstruction feature corresponding to the detection task at the first historical moment; secondly, the second initial reconstruction feature corresponding to the detection task at the current moment is randomly generated according to the task type corresponding to the detection task; finally, the first initial reconstruction feature and the second initial reconstruction feature are spliced ​​to determine the initial reconstruction feature corresponding to the current moment. In this way, on the one hand, initialization with the reconstruction feature corresponding to the detection task at the first historical moment can keep the current moment task-related information and the historical moment task-related information in a certain continuity, which helps to reduce the instability caused by random initialization; on the other hand, more diverse information can be learned by using random initialization.

[0017] Further, based on the task type corresponding to the detection task, the second initial reconstruction feature corresponding to the detection task at the current moment is randomly generated, including: based on the task type corresponding to the detection task, the initial feature corresponding to the detection task at the current moment is randomly generated; the initial feature corresponding to the task type is reconstructed to determine the second initial reconstruction feature corresponding to the detection task at the current moment.

[0018] According to the above technical means, according to the task type corresponding to the detection task, the initial features corresponding to the current moment detection task are randomly generated; the initial features corresponding to the detection task are reconstructed to determine the second initial reconstructed features corresponding to the current moment detection task. In this way, the initial features corresponding to the detection task are reconstructed to maintain consistency with the features of each historical moment.

[0019] Furthermore, based on the task type corresponding to the detection task, the initial features corresponding to the detection task at the current moment are randomly generated, including at least one of the following: when the detection task includes a moving target detection task, the coordinates in the three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, the initial features corresponding to the road segmentation detection task are determined based on the coordinates of the center position of each sub-area in the set vehicle passable area; when the detection task includes a lane line detection task, the coordinates in the two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task.

[0020] According to the above technical means, when the detection task includes a moving target detection task, the coordinates of the three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, the initial features corresponding to the road segmentation detection task are determined based on the coordinates of the center positions of each sub-area in the set vehicle passable area; when the detection task includes a lane line detection task, the coordinates of the two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task. In this way, different initialization methods are used for different detection tasks, which improves the accuracy of initialization.

[0021] Furthermore, the method of determining the first key information and the first value information corresponding to the detection task based on the panoramic view image features and the surround view image features includes: splicing the two-dimensional panoramic view features and the two-dimensional surround view features to determine the first key information corresponding to the detection task; splicing the two-dimensional panoramic view features, the three-dimensional panoramic view position features, the two-dimensional surround view features and the three-dimensional surround view position features to determine the first value information corresponding to the detection task.

[0022] According to the above technical means, the two-dimensional surrounding vision feature and the two-dimensional surround vision feature are spliced ​​to determine the first key information corresponding to the detection task; the two-dimensional surrounding vision feature, the three-dimensional surrounding vision position feature, the two-dimensional surround vision feature and the three-dimensional surround vision position feature are spliced ​​to determine the first value information corresponding to the detection task. In this way, the first value information and the first key information can include complete surrounding vision image features and surround vision image features, thereby improving the accuracy of the detection of the surrounding environment of the vehicle.

[0023] Furthermore, the first query information, the first key information and the first value information are processed using the attention mechanism to obtain target fusion features, including: using the first query information as a query matrix of the attention mechanism and the first key information as a key matrix of the attention mechanism to generate an attention matrix; and determining the target fusion features based on the attention matrix and the first value information.

[0024] According to the above technical means, the first query information is used as the query matrix of the attention mechanism, and the first key information is used as the key matrix of the attention mechanism to generate the attention matrix; based on the attention matrix and the first value information, the target fusion feature is determined. In this way, on the one hand, the attention mechanism is used to sparsely express the task-related information, the surrounding image features, and the surround image features, thereby reducing the amount of calculation and memory usage; on the other hand, the attention mechanism is used to fuse the surrounding image features and the surround image features, and to establish the association between each historical moment and the current moment, so as to improve the accuracy of the detection of the surrounding environment of the vehicle.

[0025] Furthermore, there are multiple detection tasks; based on the task type corresponding to the detection task, the first query information is determined, including: for each detection task, based on the task type corresponding to the detection task, the query information corresponding to the detection task is determined; the query information corresponding to each detection task is spliced ​​to determine the first query information.

[0026] According to the above technical means, for each detection task, based on the task type corresponding to the detection task, the query information corresponding to the detection task is determined; the query information corresponding to each detection task is spliced ​​to determine the first query information. In this way, multi-task fusion realizes comprehensive perception of the road environment and improves the accuracy and robustness of the detection of the vehicle's surrounding environment.

[0027] Furthermore, the method also includes: extracting features from the radar point cloud data of the vehicle to obtain point cloud features; determining first key information and first value information based on the surrounding image features and the surround image features, including: determining the first key information and first value information based on the surrounding image features, the surround image features and the point cloud features.

[0028] According to the above technical means, feature extraction is performed on the radar point cloud data of the vehicle to obtain point cloud features, and first key information and first value information are determined based on the surrounding view image features, surround view image features and point cloud features, so that the first key information and the first value information include purely visual surrounding view image features and surround view image features, as well as point cloud features, thereby improving the accuracy of detecting the surrounding environment of the vehicle.

[0029] A vehicle environment detection device, the environment detection device comprising:

[0030] A feature extraction module is used to extract features from the vehicle's surrounding image and surround image to obtain surrounding image features and surround image features;

[0031] A first determination module, configured to determine first query information based on a task type corresponding to the detection task;

[0032] A second determination module is used to determine the first key information and the first value information corresponding to the detection task based on the surrounding image feature and the all-around image feature;

[0033] A processing module, used to process the first query information, the first key information and the first value information by using an attention mechanism to obtain a target fusion feature;

[0034] The third determination module is used to detect the surrounding environment of the vehicle based on the target fusion feature and determine the detection result corresponding to the detection task.

[0035] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0036] An embodiment of the present application provides a vehicle, comprising the above-mentioned computer device.

[0037] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, some or all of the steps in the above method are implemented.

[0038] An embodiment of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed by a processor, some or all of the steps in the above method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the implementation process of a vehicle environment detection method proposed in this application;

[0040] Figure 2A schematic diagram of the module structure of vehicle environment detection using the sparse HAS-Propagation Transformer architecture proposed in this application;

[0041] Figure 3 A schematic diagram of a network structure for vehicle environment detection using a sparse HAS-Propagation Transformer architecture proposed in this application;

[0042] Figure 4 A schematic diagram of the specific implementation process of a BEV surround dynamic and static multi-task perception method based on a sparse HAS-Propagation Transformer architecture proposed in this application;

[0043] Figure 5 A schematic diagram of a camera coordinate system and a vehicle coordinate system proposed in this application;

[0044] Figure 6 A schematic diagram of a historical cache information structure proposed in this application;

[0045] Figure 7 A schematic diagram of historical cache Embedding reconstruction corresponding to a 3D object detection task proposed in this application;

[0046] Figure 8 A schematic diagram of historical cache Embedding reconstruction corresponding to a road segmentation detection task proposed in this application;

[0047] Fig. 9 A schematic diagram of historical cache Embedding reconstruction corresponding to a lane detection task proposed in this application;

[0048] Fig.10 A schematic diagram of a physical object for each detection task proposed in this application;

[0049] Fig.11 A detailed structural diagram of a HAS-Propagation Transformer module proposed in this application;

[0050] Fig.12 A schematic diagram of the structure of a vehicle environment detection device proposed in this application;

[0051] Fig.13 A hardware entity schematic diagram of a computer device proposed in this application. DETAILED DESCRIPTION

[0052] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0053] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0054] The present application embodiment proposes a vehicle environment detection method, such as Figure 1 As shown, the vehicle environment detection method includes the following steps S101 to S105, wherein:

[0055] Step S101: extracting features from the surrounding view image and surround view image of the vehicle respectively to obtain surrounding view image features and surround view image features;

[0056] Here, a panoramic image is an image that can show the surrounding environment of a certain area or object. This image can be a visual presentation that radiates from a central point. A panoramic image can capture high-definition details of the surrounding environment, but the coverage is limited.

[0057] Here, surround view imaging can capture and display a 360-degree panoramic image of a scene or object, with a large coverage area but large distortion.

[0058] In some embodiments, the surrounding image is acquired by a surrounding camera, such as a pinhole perspective camera; and the surrounding image is acquired by a surround-view camera, such as a fisheye camera.

[0059] In some embodiments, the surrounding view image feature includes a two-dimensional surrounding view feature and a three-dimensional surround view position feature.

[0060] In some implementations, the surround view image feature includes a two-dimensional surround view feature and a three-dimensional surround view position feature.

[0061] Step S102: determining first query information based on the task type corresponding to the detection task;

[0062] In some implementations, the detection tasks may include a moving target detection task, a road segmentation detection task, and a lane line detection task.

[0063] In some implementations, different detection tasks are of different task types.

[0064] In some implementations, the first query information is information related to the detection task.

[0065] In some implementations, a query matrix, a key matrix, and a value matrix may be determined according to a task type corresponding to the detection task, and the first query information may be determined using a hybrid attention mechanism.

[0066] Step S103: determining first key information and first value information based on the surrounding view image feature and the all-around view image feature;

[0067] In some embodiments, the first key information includes a two-dimensional surrounding view feature and a two-dimensional surround view feature.

[0068] In some embodiments, the first value information includes a two-dimensional surrounding viewing feature, a three-dimensional surrounding viewing position feature, a two-dimensional surround viewing feature, and a three-dimensional surround viewing position feature.

[0069] Step S104: using an attention mechanism to process the first query information, the first key information and the first value information to obtain a target fusion feature;

[0070] In some embodiments, the first query information is used as a query matrix, the first key information is used as a key matrix, and the first value information is used as a value matrix, and a periscopic attention mechanism is used to determine target fusion features.

[0071] Step S105: Based on the target fusion features, the surrounding environment of the vehicle is detected to determine the detection result corresponding to the detection task.

[0072] In some embodiments, the target fusion feature corresponding to the detection task is connected to the corresponding decoding head network to obtain multiple decoding results corresponding to the detection task, and the decoding results are sorted according to the decoding confidence corresponding to the decoding results, and the target number of target decoding results are selected from the multiple decoding results; based on the target decoding results, the detection result corresponding to the detection task is determined.

[0073] In some implementations, the sorted decoding results are stored in a history cache queue using a first-in-first-out (FIFO) principle for processing at the next moment.

[0074] In some embodiments, the decoding of the motion target detection task can adopt a simple multi-layer perceptron network with an input dimension of C and an output dimension that aligns information such as target confidence, target category, center position, length, width and height of the rectangular box, target direction and target speed, which is set according to actual needs.

[0075] In some embodiments, the decoding of the road segmentation detection task requires segmenting the pixel-level position area on the BEV plane. Since each road query (Query) corresponds to the bird's-eye view (BEV) plane S H ×S W One of the rectangular patches, each patch corresponds to r h × w Pixel size, so the input dimension is C and the output dimension is r h × w A simple multi-layer perceptron network with 1-dimensional confidence can also be used to calculate r h × w Each patch is segmented into pixels.

[0076] In some embodiments, the lane line detection task uses a multi-layer perceptron network with an input dimension of C, but its output dimension depends on the vectorized expression of the lane line. Typical expressions include MapTR, PETRv2, and even 3D lane line PersFormer (such as 3D lane lines). It can be measured in combination with actual needs and complexity constraints, and this application does not impose any restrictions.

[0077] In an embodiment of the present application, first, feature extraction is performed on the surrounding view image and the surround view image of the vehicle respectively to obtain the surrounding view image feature and the surround view image feature; secondly, the first query information is determined according to the task type corresponding to the detection task, and the first key information and the first value information corresponding to the detection task are determined according to the surrounding view image feature and the surround view image feature; the first query information, the first key information and the first value information are processed by using the attention mechanism to obtain the target fusion feature; finally, according to the target fusion feature, the surrounding environment of the vehicle is detected to determine the detection result corresponding to the detection task. In this way, on the one hand, the attention mechanism is used to integrate the input and output of the surrounding view image features and the surround view image features, which reduces the resource occupation and delayed waiting caused by independent operation, and effectively improves the information processing efficiency. Moreover, by constructing the key information and value information in the attention mechanism with the surrounding view image features and the surround view image features, the surrounding view image features and the surround view image features can be better integrated and utilized, thereby improving the perception ability in the process of environmental detection and improving the accuracy of environmental detection; on the other hand, the attention mechanism is used to sparsely express the task-related information, surrounding view image features, and surround view image features, thereby reducing the amount of computation and memory usage in the feature fusion process.

[0078] In some embodiments, the first key information and the first value information can be determined based only on the surrounding image features and the surround image features. In this way, the target fusion features for detecting the surrounding environment of the vehicle can be obtained by fusing the pure visual surrounding image features and the surround image features, thereby reducing the high cost of using other non-visual acquisition equipment (such as laser radar, etc.) for detection, thereby reducing the cost of detecting the surrounding environment of the vehicle.

[0079] In some embodiments, the first key information and the first value information can be determined based on the surrounding image features, the surround image features, and the non-visual modality features obtained by extracting the non-visual modality perception data of the vehicle (such as radar point cloud data, etc.). In this way, the surrounding image features and the surround image features under the visual modality, as well as the non-visual modality features, can be integrated to improve the perception ability of the environment, thereby improving the accuracy of the detection of the surrounding environment of the vehicle.

[0080] In some embodiments, the surrounding view image features include two-dimensional surrounding view features and three-dimensional surrounding view position features, and the surround view image features include two-dimensional surround view features and three-dimensional surround view position features; the step S101 of extracting features from the surrounding view image and the surround view image of the vehicle to obtain the surrounding view image features and the surround view image features may include the following steps S1011 and S1012, wherein:

[0081] Step S1011: performing two-dimensional coding on the surrounding view image and the all-around view image to determine the two-dimensional surrounding view feature and the two-dimensional all-around view feature;

[0082] In some implementations, two independent 2D convolutional coding backbone networks are used to perform feature encoding on the surrounding view image and the surround view image, respectively, to obtain two-dimensional surrounding view features and two-dimensional surround view features.

[0083] In some implementations, the backbone network may generally adopt, but is not limited to, a series of commonly used 2D convolutional network structures with a certain depth level such as ResNet, EfficientNet, SwinTransformer, VoVNetV2, etc.

[0084] In some embodiments, all surrounding view images and all surround view images each share a set of network weights.

[0085] Step S1012: performing three-dimensional coding on the surrounding view image and the all-around view image to determine the three-dimensional surrounding view position feature and the three-dimensional all-around view position feature.

[0086] In some embodiments, first, according to the distortion model of the panoramic camera and the distortion parameters corresponding to the distortion model, combined with the internal parameters of the panoramic camera, the de-distorted true vector direction corresponding to each pixel of the panoramic image in the panoramic camera coordinate system is calculated; then, reasonable equally spaced depth segments are set in the BEV space, and the coordinate values ​​of each pixel of the image in different depth segments in the panoramic camera coordinate system are constructed in combination with the vector direction of each pixel; then, the coordinates of each pixel in different depth segments are converted to the vehicle coordinate system by using the external parameter rotation and displacement information of the panoramic camera relative to the vehicle coordinate system; finally, the reference positional encoding (RefPE) is used to encode the coordinates of different depth segments corresponding to each pixel of the panoramic image in the vehicle coordinate system to obtain three-dimensional panoramic position features.

[0087] In some embodiments, first, according to the distortion model of the surround view camera and the distortion parameters corresponding to the distortion model, combined with the internal parameters of the surround view camera, the de-distorted true vector direction corresponding to each pixel of the surround view image in the surround view camera coordinate system is calculated; then, reasonable equally spaced depth segments are set in the BEV space, and the coordinate values ​​of each pixel of the image in different depth segments in the surround view camera coordinate system are constructed in combination with the vector direction of each pixel; then, the coordinates of each pixel in different depth segments are converted to the vehicle coordinate system by using the external parameter rotation and displacement information of the surround view camera relative to the vehicle coordinate system; finally, the reference positional encoding (RefPE) is used to encode the coordinates of different depth segments corresponding to each pixel of the surround view image in the vehicle coordinate system to obtain three-dimensional surround position features.

[0088] In this embodiment, the surrounding image features include two-dimensional surrounding features and three-dimensional surrounding position features, and the surround image features include two-dimensional surround features and three-dimensional surround position features; the surrounding image and the surround image are two-dimensionally encoded to determine the two-dimensional surrounding features and the two-dimensional surround features; the surrounding image and the surround image are three-dimensionally encoded to determine the three-dimensional surrounding position features and the three-dimensional surround position features. In this way, the surrounding image features and the surround image features can include complete two-dimensional image information and three-dimensional spatial position information, thereby improving the accuracy of environmental detection.

[0089] In some embodiments, the step S102 of determining the first query information based on the task type corresponding to the detection task may include the following steps S1021 and S1022, wherein:

[0090] Step S1021: determining second query information, second key information, and second value information corresponding to the detection task based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment;

[0091] In some implementations, the initial features at the current moment are initialized according to the task type corresponding to the detection task, and the second query information corresponding to the detection task is determined according to the detection result corresponding to the detection task at the previous moment and the initial features at the current moment.

[0092] In some implementations, the second key information and the second value information are determined based on the initial features of the current moment and the detection results corresponding to the detection tasks at each historical moment; the second key information and the second value information are consistent.

[0093] Step S1022: Utilize the attention mechanism to process the second query information, the second key information and the second value information to obtain the first query information.

[0094] In some embodiments, the second query information is used as a query matrix, the second key information is used as a key matrix, and the second value information is used as a value matrix, which are input into a hybrid attention mechanism to output the first query information.

[0095] In this embodiment, the second query information, second key information and second value information corresponding to the detection task are determined according to the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment; the second query information, second key information and second value information are processed by using the attention mechanism to obtain the first query information, so that the task-related information at the current moment and the task-related information at each historical moment can be integrated.

[0096] In some embodiments, the step S1021 of determining the second query information, the second key information, and the second value information corresponding to the detection task based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment may include the following steps S10211 to S10214, wherein:

[0097] Step S10211: reconstructing the detection results corresponding to the detection tasks at each of the historical moments based on the posture conversion relationship between the vehicle at each of the historical moments and the current moment, and determining the reconstructed features corresponding to the detection tasks at each of the historical moments;

[0098] In some implementations, the decoding results corresponding to each historical moment and the detection task are read from the historical cache queue, and the decoding results corresponding to each historical moment and the detection task are aligned to the current moment to obtain the updated decoding results corresponding to the detection tasks at each historical moment at the current moment;

[0099] In some implementations, the updated decoding results corresponding to the detection tasks at each historical moment are reconstructed to generate high-dimensional reconstructed Embedding vectors corresponding to each historical moment that are spatially aligned with the current moment.

[0100] Step S10212: determining an initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task and the reconstruction feature corresponding to the detection task at the first historical moment; the first historical moment is the moment before the current moment;

[0101] In some implementations, the reconstruction feature corresponding to the detection task at the first historical moment is propagated to initialize the initial reconstruction feature at the current moment, thereby obtaining the first initial reconstruction feature.

[0102] In some implementations, a plurality of second initial reconstruction features are generated by random initialization according to the task type corresponding to the detection task.

[0103] In some implementations, the first initial reconstructed feature and the second initial reconstructed feature are concatenated to obtain an initial reconstructed feature corresponding to the detection task at the current moment.

[0104] In some embodiments, the initial reconstructed features corresponding to the moving target detection task include the initial position of the moving target in the BEV space and the context features; the initial reconstructed features corresponding to the road segmentation detection task include the BEV ground plane S H ×S W The context features corresponding to the rectangular patches of the same size are fixed at the center points of each patch; the initial reconstructed features corresponding to the lane line detection task include the initial geometric center position and context features of each lane line or curb.

[0105] Step S10213: determining the initial reconstruction feature corresponding to the detection task at the current moment as the second query information;

[0106] Step S10214: Determine the second key information and the second value information based on the reconstruction features corresponding to the detection task at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment; the second historical moment is a historical moment other than the first historical moment.

[0107] In some implementations, the reconstructed features corresponding to each second historical moment detection task and the initial reconstructed features corresponding to the current moment detection task are concatenated to obtain second key information and second value information; the second key information and the second value information are consistent.

[0108] In this embodiment, firstly, based on the posture conversion relationship between each historical moment and the vehicle at the current moment, the detection results corresponding to the detection tasks at each historical moment are reconstructed, the reconstruction features corresponding to the detection tasks at each historical moment are determined, and the initial reconstruction features corresponding to the detection tasks at the current moment are determined according to the task type corresponding to the detection task and the reconstruction features corresponding to the detection tasks at the first historical moment; the first historical moment is the previous moment of the current moment; secondly, the initial reconstruction features corresponding to the detection tasks at the current moment are determined as the second query information; finally, based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection tasks at the current moment, the second key information and the second value information are determined; the second historical moment is the historical moment other than the first historical moment. In this way, the second query information includes the initial reconstruction features corresponding to the detection tasks at the current moment, so that the task-related information at the current moment and the task-related information at each historical moment can be fused in the first query information, and then the association between the task-related information at the current moment and the task-related information at each historical moment can be established in the target fusion feature, so as to further improve the accuracy of environmental detection.

[0109] In some embodiments, the step S10211 described in step S10211 reconstructs the detection results corresponding to the detection tasks at each of the historical moments based on the posture conversion relationship between the vehicle at each of the historical moments and the current moment, and determines the reconstructed features corresponding to the detection tasks at each of the historical moments, which may include the following steps S102111 and S102112, wherein:

[0110] Step S102111: Based on the posture conversion relationship, align the detection results corresponding to the detection tasks at each of the historical moments to the current moment, and determine the updated detection results corresponding to the detection tasks at each of the historical moments;

[0111] In some implementations, based on the posture conversion relationship, the detection results corresponding to the moving target detection task, the road segmentation detection task, and the lane line detection task stored at each historical moment are converted and aligned to the vehicle coordinate system at the current moment.

[0112] It is understandable that since the dimension of the center position corresponding to the road segmentation detection task and the lane line detection task is only 2, it is necessary to supplement the height value of the ground in the vehicle coordinate system in the height Z-axis direction to expand it to 3 dimensions, and then calculate the coordinates of the center position corresponding to the road segmentation detection task and the lane line detection task at each historical moment to align to the current moment according to the conversion formula.

[0113] Step S102112: Based on the posture conversion relationship, the time difference between each historical moment and the current moment, and the movement speed corresponding to the detection task, reconstruct the updated detection results corresponding to the detection task at each historical moment, and determine the reconstructed features corresponding to the detection task at each historical moment.

[0114] In some embodiments, the pose conversion relationship, time difference and target motion speed are used to calculate the MLN corresponding coefficient according to the motion-aware regularization (MLN) method, and the MLN corresponding coefficient is used to align each detection result cached at each historical moment to the current moment Embedding for reconstruction.

[0115] In this embodiment, the detection results corresponding to the detection tasks at each historical moment are aligned to the current moment according to the posture conversion relationship, and the updated detection results corresponding to the detection tasks at each historical moment are determined; according to the posture conversion relationship, the time difference between each historical moment and the current moment, and the movement speed corresponding to the detection task, the updated detection results corresponding to the detection tasks at each historical moment are reconstructed to determine the reconstructed features corresponding to the detection tasks at each historical moment. In this way, the detection results corresponding to the detection tasks at the historical moment are aligned to the current moment and reconstructed, so that the information related to the current moment task and the information related to the historical moment task maintain continuity and consistency in semantics.

[0116] In some embodiments, the step S10212 of determining the initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task and the reconstruction feature corresponding to the detection task at the first historical moment may include the following steps S102121 to S102123, wherein:

[0117] Step S102121: determining a first initial reconstruction feature corresponding to the detection task at the current moment based on the reconstruction feature corresponding to the detection task at the first historical moment;

[0118] In some implementations, the reconstructed features corresponding to the detection task at the first historical moment are propagated to obtain the first initial reconstructed features corresponding to the detection task at the current moment.

[0119] Step S102122: randomly generating a second initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task;

[0120] Step S102123: concatenate the first initial reconstruction feature and the second initial reconstruction feature to determine the initial reconstruction feature corresponding to the current moment.

[0121] In some implementations, the moving object detection task and the lane line detection task directly concatenate the first initial reconstructed feature and the second initial reconstructed feature to obtain the initial reconstructed feature.

[0122] In some embodiments, after the road segmentation detection task splices the first initial reconstruction feature with the second initial reconstruction feature, it is also necessary to confirm that the Embedding reconstructed at the first historical moment corresponds to the 2D ground plane S of the BEV space in the current vehicle coordinate system. H ×S W The location of the rectangular patch.

[0123] In this embodiment, first, the first initial reconstruction feature corresponding to the detection task at the current moment is determined based on the reconstruction feature corresponding to the detection task at the first historical moment; secondly, the second initial reconstruction feature corresponding to the detection task at the current moment is randomly generated based on the task type corresponding to the detection task; finally, the first initial reconstruction feature and the second initial reconstruction feature are spliced ​​to determine the initial reconstruction feature corresponding to the current moment. In this way, on the one hand, using the reconstruction feature initialization corresponding to the detection task at the first historical moment can keep the current moment task related information and the historical moment task related information continuous to a certain extent, which helps to reduce the instability caused by random initialization; on the other hand, more diverse information can be learned using random initialization.

[0124] In some embodiments, the step S102122 of randomly generating the second initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task may include the following steps S1021221 and S1021222, wherein:

[0125] Step S1021221: Based on the task type corresponding to the detection task, randomly generate an initial feature corresponding to the detection task at the current moment;

[0126] In some implementations, initial features corresponding to the detection task are generated according to different task types.

[0127] In some implementations, the initial value of the context Embedding vector of each detection task is set to a 0 vector.

[0128] Step S1021222: reconstruct the initial features corresponding to the detection task to determine the second initial reconstructed features corresponding to the detection task at the current moment.

[0129] In this embodiment, according to the task type corresponding to the detection task, the initial features corresponding to the current moment detection task are randomly generated; the initial features corresponding to the detection task are reconstructed to determine the second initial reconstructed features corresponding to the current moment detection task. In this way, the initial features corresponding to the detection task are reconstructed to maintain consistency with the features of each historical moment.

[0130] In some embodiments, the step S1021221 of randomly generating the initial features corresponding to the detection task at the current moment based on the task type corresponding to the detection task may include at least one of the following steps S10212211 to S1021223:

[0131] Step S1611: when the detection task includes a moving target detection task, uniformly sampling the coordinates of the three-dimensional space to determine the initial features corresponding to the moving target detection task;

[0132] In some implementations, the center position corresponding to the moving target detection task is initialized using uniformly distributed coordinates of the position in the 3D space under the BEV space under the vehicle coordinate system.

[0133] Step S1612: when the detection task includes a road segmentation detection task, determining the initial features corresponding to the road segmentation detection task based on the coordinates of the center positions of each sub-area in the set vehicle passable area;

[0134] In some implementations, the center position corresponding to the road segmentation detection task is initialized using S H ×S W The coordinates of the center position of the remaining patch in a rectangular patch (i.e., sub-region) after Propagation at the first historical moment.

[0135] Step S1613: When the detection task includes a lane line detection task, uniformly sample the coordinates of the two-dimensional space to determine the initial features corresponding to the lane line detection task.

[0136] In some implementations, the center position of the lane detection task is initialized using uniformly distributed coordinates on a 2D ground plane in the BEV space in the vehicle coordinate system.

[0137] In this embodiment, when the detection task includes a moving target detection task, the coordinates of the three-dimensional space are uniformly sampled to determine the initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, the initial features corresponding to the road segmentation detection task are determined based on the coordinates of the center positions of each sub-area in the set vehicle passable area; when the detection task includes a lane line detection task, the coordinates of the two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task. In this way, different initialization methods are used for different detection tasks, which improves the accuracy of initialization.

[0138] In some embodiments, the step S103 of determining the first key information and the first value information based on the surrounding image feature and the all-around image feature may include the following steps S1031 and S1032, wherein:

[0139] Step S1031: splicing the two-dimensional surrounding vision feature and the two-dimensional surround vision feature to determine the first key information corresponding to the detection task;

[0140] Step S1032: splicing the two-dimensional surrounding vision feature, the three-dimensional surrounding vision position feature, the two-dimensional surround vision feature and the three-dimensional surround vision position feature to determine the first value information corresponding to the detection task.

[0141] In this embodiment, the two-dimensional surrounding vision feature and the two-dimensional surround vision feature are spliced ​​to determine the first key information corresponding to the detection task; the two-dimensional surrounding vision feature, the three-dimensional surrounding vision position feature, the two-dimensional surround vision feature and the three-dimensional surround vision position feature are spliced ​​to determine the first value information corresponding to the detection task. In this way, the first value information and the first key information can include complete surrounding vision image features and surround vision image features, thereby improving the accuracy of the detection of the surrounding environment of the vehicle.

[0142] In some embodiments, the step S104 of using the attention mechanism to process the first query information, the first key information, and the first value information to obtain the target fusion feature may include the following steps S1041 and S1042, wherein:

[0143] Step S1041: using the first query information as a query matrix of the attention mechanism and the first key information as a key matrix of the attention mechanism to generate an attention matrix;

[0144] Step S1042: Determine the target fusion feature based on the attention matrix and the first value information.

[0145] In this embodiment, the first query information is used as the query matrix of the attention mechanism, and the first key information is used as the key matrix of the attention mechanism to generate the attention matrix; based on the attention matrix and the first value information, the target fusion feature is determined. In this way, on the one hand, the attention mechanism is used to sparsely express the task-related information, the surrounding image features, and the surround image features, thereby reducing the amount of calculation and memory usage; on the other hand, the attention mechanism is used to fuse the surrounding image features and the surround image features, and to establish the association between each historical moment and the current moment, so as to improve the accuracy of the detection of the surrounding environment of the vehicle.

[0146] In some embodiments, the number of the detection tasks is multiple; the step S102 of determining the first query information based on the task type corresponding to the detection task may include the following steps S1023 and S1024, wherein:

[0147] Step S1023: for each of the detection tasks, based on the task type corresponding to the detection task, determining query information corresponding to the detection task;

[0148] In some implementations, the detection task may include at least one of a moving target detection task, a road segmentation detection task, a lane line detection task, and the like.

[0149] In some implementations, each detection task corresponds to different query information.

[0150] Step S1024: concatenate the query information corresponding to each of the detection tasks to determine the first query information.

[0151] In this embodiment, for each detection task, based on the task type corresponding to the detection task, the query information corresponding to the detection task is determined; the query information corresponding to each detection task is spliced ​​to determine the first query information. In this way, multi-task fusion realizes comprehensive perception of the road environment and improves the accuracy and robustness of the detection of the vehicle's surrounding environment.

[0152] In some embodiments, the above method may further include the following step S106, wherein:

[0153] Step S106: extracting features from the radar point cloud data of the vehicle to obtain point cloud features;

[0154] In some embodiments, radar point cloud data is obtained by scanning the vehicle's surroundings with a lidar and collecting reflected signals. The radar point cloud data provides high-precision three-dimensional perception information of the vehicle's surroundings, can accurately reflect the 3D spatial depth position of objects and the environment, and has incomparable advantages over vision in ranging and precise positioning of spatial objects, especially for sudden intrusions or unknown obstacles not on the whitelist, and has detection capabilities superior to vision.

[0155] The step S103 described above of determining the first key information and the first value information based on the surrounding image feature and the all-around image feature may include the following step S1033, wherein:

[0156] Step S1033: Determine first key information and first value information based on the surrounding view image feature, the all-around view image feature and the point cloud feature.

[0157] In some embodiments, the purely visual surrounding view image features and surround view image features are fused with the point cloud features to improve the accuracy of the vehicle as an environment detection, that is, the surrounding view image features, surround view image features and point cloud features can be spliced ​​to obtain the first key information and the first value information.

[0158] In some embodiments, the two-dimensional panoramic view features, the two-dimensional surround view features and the point cloud features are spliced ​​to determine the first key information corresponding to the detection task; the two-dimensional panoramic view features, the three-dimensional surround view position features, the two-dimensional surround view features, the three-dimensional surround view position features and the point cloud features are spliced ​​to determine the first value information corresponding to the detection task.

[0159] In this embodiment, feature extraction is performed on the radar point cloud data of the vehicle to obtain point cloud features, and first key information and first value information are determined based on the surrounding view image features, surround view image features and point cloud features, so that the first key information and the first value information include purely visual surrounding view image features and surround view image features, as well as point cloud features, thereby improving the accuracy of detection of the vehicle's surrounding environment.

[0160] The following describes the application of the embodiments of the present application in actual scenarios.

[0161] With the rapid development of autonomous driving technology, the current perception method based on multi-source and multi-modal BEV fusion has become the mainstream technical method for driving and parking perception, and is widely favored by the industry. BEV perception uses multi-source and multi-modal information such as cameras and radars around the vehicle. By uniformly mapping the feature information of each sensor to the BEV space, it realizes the fusion and coverage of all-round feature information of the vehicle in front, behind, left, right, far, medium and near distances, thereby greatly improving the perception of important external targets and the overall road topology. This perception method, which is different from a single perspective, directly performs spatial coverage information fusion of different sources and different modes at the feature layer of each sensor data. It can not only avoid the interference of single sensor truncation of targets and the loss of useful information caused by post-fusion, but also greatly enhance the ability to express spatial features, thereby effectively improving the accuracy of global environmental perception and detection.

[0162] In multimodal BEV perception, LiDAR can accurately reflect the 3D spatial depth position of objects and the environment, and has incomparable advantages in ranging and precise positioning of spatial objects, especially for sudden intrusion or unknown obstacle avoidance on the non-whitelist, which has better detection capabilities than vision. However, the high cost of LiDAR has greatly limited the promotion of BEV perception methods based on radar fusion, so that it can only be adapted to high-end models at present, and it is difficult to penetrate into the low-end and mid-end models on a large scale. In view of this, low-cost sensor fusion solutions represented by pure vision are more cost-effective for most companies in the industry, providing the possibility of large-scale market expansion. Visual image information has the characteristics of rich details and full features, which is sufficient to fully reflect the color, shape, texture, posture and other multi-dimensional information of the surrounding environment scene and targeted targets of the vehicle, and has always been regarded as the optimal data source for classification recognition and scene semantic representation. However, the disadvantages of relying solely on visual image content are also prominent, that is, it only reflects 2D plane information, unlike the laser point cloud, which naturally has good depth of field properties, which is destined to be difficult to achieve high-precision measurement and positioning. In this regard, although the current academic community has proposed a variety of methods using deep neural networks to improve the prediction accuracy of visual depth of field, the current depth of field prediction for images is still coarse-grained. How to achieve accurate positioning of targets in 2D images in the 3D physical world still poses a major challenge in this research field.

[0163] In view of the above problems, without considering the lidar sensor, one of the most intuitive ways to alleviate them is to make full use of the image information from various perspectives on the vehicle to improve the comprehensive perception and positioning capabilities of the target and environment. Under the premise of supporting the driving and parking function requirements, autonomous driving vehicles will be equipped with panoramic and surround view cameras that can cover a 360° viewing angle around the vehicle to achieve detection coverage around the vehicle, especially at medium and short distances. However, in terms of mass production applications, due to the large differences between the panoramic image and the surround view image in terms of field of view (FOV), distortion mode, etc., the panoramic view and surround view are generally processed independently for driving and parking tasks. There is a certain technical difficulty in truly realizing fusion at the feature level, and most of the output information of the two is only fused and utilized in the post-processing stage. This method brings two problems. On the one hand, the two independent models will increase the consumption of limited resources on the vehicle side and the high latency caused by preemption. On the other hand, the inherent blind spot feature defects of the panoramic view cannot be effectively compensated. Therefore, it is necessary to build an effective integrated network structure to fully and efficiently integrate the panoramic and surround image information to form a true multi-view and multi-source perception input and output, which can not only significantly solve the aforementioned problems, but also bring about a further improvement in the perception and positioning accuracy.

[0164] In recent years, a large number of research practices in the field of natural language processing and computer vision have shown that compared with the CNN structure with limited visual receptive field, the neural network architecture based on the attention mechanism (Transformer network architecture) has more advantages in building global and local feature associations, homologous and non-homologous task associations, etc., and can greatly improve the prediction and judgment capabilities of complex scenes. Based on this feature, the advantages of the Transformer architecture are fully utilized, and through the Attention mechanism and high-efficiency and low-overhead sparse query expression, the abstract association relationship between the predicted object and the surrounding image information, the dynamic target and the static road structure information, and the current moment and the historical information in the BEV perception network is established, realizing full association calculation across full-view cameras and across all task contents, which helps to significantly improve the model's perception and prediction accuracy of the overall complex environment while ensuring operational efficiency, and has good implementation feasibility.

[0165] In the related technology, the multi-road surround view image BEV perception task under the pinhole camera model only corresponds to the output of 3D target information, and cannot realize the fusion and utilization of the surround view fisheye camera image. It also does not support multi-task output such as road information, and does not consider the correlation between dynamic and static targets. The usage scenarios are relatively limited.

[0166] In addition, in the related art, a number of images are obtained from multiple fisheye cameras, and features are extracted from the current frame fisheye image and the historical frame fisheye image to obtain the first feature map corresponding to the current frame fisheye image and the second feature map corresponding to the historical frame fisheye image, and then the BEV merged features are obtained based on the first feature map and the second feature map, and finally the target tracking detection is performed based on the BEV merged features. This method only performs feature extraction and mapping processing on fisheye images, and fisheye distortion will inevitably have a significant negative impact on the final recognition and precise positioning, and the fisheye surround detection distance is limited. At the same time, the model cannot reuse conventional pinhole camera images, and the application scenarios are also very limited.

[0167] Recently, related technologies have proposed a sparse query design based on the Transformer structure, a time mechanism centered on dynamic target objects, an online model execution, long-term historical cache information propagation through frame-by-frame object queries, and the introduction of a motion perception layer normalization to model the motion of objects. Although this method also builds a sparse BEV perception architecture based on Transformer, it only considers traditional panoramic image input, and does not have a fusion mechanism for surround fisheye image information. The output task is only a specific 3D dynamic target, and does not have the ability to perceive the static structure of the road. Naturally, there is no way to build the correlation between important environmental elements such as surround, surround, dynamic, and static, making it difficult to cope with complex multi-task scenarios.

[0168] Based on the above description, the embodiment of the present application provides a BEV surround view dynamic and static multi-task perception method based on a sparse HAS-Propagation Transformer architecture, which realizes the integrated perception multi-task processing output of the surround view and surround view images covering 360° around the vehicle by constructing a set of Transformer model architectures based on time-series sparse feature expression. This Transformer model architecture relies solely on the image information input by the vehicle's surrounding visual sensors to solve the dynamic and static target and environment perception problems encountered during parking and driving, thereby reducing the cost of detection sensors. The integrated input and output structure of surround view and surround view, on the one hand, avoids the resource grabbing and delayed waiting caused by the independent operation of the traditional surround view and surround view perception tasks, effectively improving the information processing efficiency; on the other hand, by fusing and utilizing the surround view and surround view information, the purpose of greatly improving the perception accuracy can be achieved. The dynamic and static perception multi-task sparse feature expression adopted by the Transformer architecture can not only effectively improve the large amount of memory consumption and delay degradation caused by the generation of intensive BEV feature maps that are widely used today, but also the Hybrid Attention and AS Attention mechanisms in the HAS-Propagation Transformer module realize the abstract feature association between dynamic targets and static road structures, between road passable areas and roadside lanes, and between current states and historical states, thereby making the positional topological relationship between the output dynamic targets and the static road environment more accurate. In summary, this embodiment fully utilizes the low-cost visual sensor information of the vehicle, and constructs an effective association relationship between temporal dynamic and static targets based on the Transformer to achieve efficient perception output with integrated input and output.

[0169] Figure 2 A schematic diagram of the module structure of a vehicle environment detection using a sparse HAS-Propagation Transformer architecture proposed in this application is shown in FIG. Figure 2As shown, the module structure includes an inference module 200 and a training module 240, wherein the modules in the inference module 200 are as follows: a surrounding image is acquired in a surrounding pinhole camera imaging module 201, a surrounding image is 2D encoded in a surrounding encoding network 2D image feature encoding module 202, and a surrounding imaging 3D pseudo point cloud module 203 and a surrounding 3D position encoding module 204 complete 3D encoding of the surrounding image; a surrounding fisheye camera imaging module 205 acquires a surrounding image, a surrounding image is 2D encoded in a surrounding encoding network 2D image encoding module 206, and a surrounding imaging 3D pseudo point cloud 207 and a surrounding 3D position encoding module 208 complete 3D encoding of the surrounding image 3D encoding of paired surround view images; caching information of historical moments in the historical cache information module 212 according to the vehicle body positioning posture and timestamp information 214, and generating historical Embedding reconstruction vectors in the historical Embedding reconstruction generation module 211 according to the historical cache information; initializing the current Query in the current Query initialization module 213 using the historical Embedding reconstruction vectors; inputting the 2D encoding features, 3D encoding features of the surround view images, the 2D encoding features, 3D encoding features of the surround view images, the historical Embedding reconstruction vectors, and the current Query initialization obtained by the above modules into the HAS Propagation Transformer network module 220, and decoding the features output by the HAS Propagation Transformer network module 220 in the 3D target detection task decoding module 231, the road segmentation detection task decoding module 232, and the lane line detection task decoding module 233, respectively. The modules in the training module 240 are as follows: the current query initialization module 213 is trained using the initial query optimization iteration 241; the 3D target detection task decoding module 231 is trained using the 3D target annotation true value information 242; the road segmentation detection task decoding module 232 and the lane line detection task decoding module 233 are trained using the road structured annotation true value information 243.

[0170] Figure 3 FIG. 4 shows a schematic diagram of a network structure for vehicle environment detection using a sparse HAS-Propagation Transformer architecture proposed in this application. Figure 3 As shown, the panoramic coding network 301 is used to encode the current frame panoramic images 1 to N a Perform 2D encoding to obtain panoramic features The panoramic image is 3D encoded using the panoramic 3D position code 302 to obtain the panoramic 3D code The surround view coding network 303 is used to surround the current frame images 1 to N s Perform 2D encoding to obtain surround features The surround view image is 3D encoded using the surround view 3D position encoding 304 to obtain the surround view 3D encoding Initialize the Embedding reconstruction vector 307 of the previous frame in the historical cache queue 305 to obtain the current frame initialization Query information 308; align the information in the historical cache queue 305 with the current frame time and space, and reconstruct it to obtain the historical frame Embedding reconstruction vector 306; Panoramic 3D coding Look around feature Surround 3D Encoding The current frame initialization Query information 308 and the historical frame Embedding reconstruction vector 306 are input into the HAS-Propagation Transformer network module 220, and the 3D object detection task Query 309, the road segmentation detection task Query 310, and the lane line detection task Query 311 are output. The 3D object detection decoding task header 312, the road segmentation detection decoding task header 313, and the lane line detection decoding task header 314 are used to decode the 3D object detection task Query 309, the road segmentation detection task Query 310, and the lane line detection task Query 311 respectively to obtain the current frame Top-K cache 315; the current frame Top-K cache 315 is stored in the historical cache queue 305.

[0171] The specific implementation process of the BEV surround dynamic and static multi-task perception method of the sparse HAS-Propagation Transformer architecture in the embodiment of the present application is as follows Figure 4 As shown, it may include steps S401 to S406:

[0172] Step S401: Panoramic and surround image: 2D feature coding and 3D position coding;

[0173] The coordinate system provided in this application is Figure 5 As shown, it includes a camera coordinate system 501 and a vehicle coordinate system 502 .

[0174] Firstly, the image of the panoramic pinhole camera and the image of the panoramic fisheye camera are encoded using two independent 2D convolutional coding backbone networks to obtain the 2D coding features of the panoramic image. and surround image 2D encoding features Then, the periscopic camera intrinsic parameters and its distortion parameter model, the surround camera intrinsic parameters and its distortion parameter model are used to construct the frustum pseudo point cloud of each camera image after dedistortion with a certain depth interval, and the corresponding camera extrinsic parameters are combined to transform its coordinates from the camera coordinate system 501 to the vehicle coordinate system 502, completing the 3D position encoding corresponding to each pixel point of the periscopic image. 3D position encoding corresponding to each pixel of the surround image

[0175] In some embodiments, the above step S401 may include the following steps S4011 and S4012:

[0176] Step S4011: Perform 2D feature encoding on the panoramic camera image and the surround camera image The backbone network can usually adopt but is not limited to a series of commonly used 2D convolutional network structures with a certain depth level such as ResNet, EfficientNet, SwinTransformer, VoVNetV2, etc., and all surrounding images and all surround images each share a set of network weights.

[0177] Step S4012: Perform 3D position encoding on the panoramic camera image and the surround camera image The specific operation method is as follows: for a camera image of a certain viewing angle:

[0178] First, according to the distortion model and distortion parameters of the panoramic or surround camera, combined with the corresponding camera internal parameters Calculate the dedistorted true vector direction corresponding to each pixel of the restored image in the camera coordinate system.

[0179] Next, the physical space size H is detected according to the BEV to be generated. B ×W B , set reasonable equally spaced depth segments {z k |k∈n}, and combined with the true vector direction of each pixel, the coordinate value of each pixel (i, j) in the image at different depth segments in the camera coordinate system is constructed

[0180] Then, the corresponding camera is rotated relative to the vehicle coordinate system using the external parameters and displacement Information, convert the coordinates of each pixel in different depth segments to the vehicle coordinate system

[0181] Finally, RefPE encoding is used to encode the coordinates of different depth segments corresponding to each pixel point in the vehicle coordinate system. The specific encoding method is shown in formula (1) and formula (2):

[0182]

[0183] in, ξ represents the Fourier position code, s i,j,kIt represents the weighted weight of the depth segment k corresponding to the pixel (i, j), which can be set manually or obtained through training. Concat and MLP represent the concatenation operation and the multi-layer perceptron network respectively.

[0184] Step S402: historical cache information Embedding is reconstructed and generated;

[0185] In some embodiments, cache information about dynamic and static task expressions such as 3D target detection tasks (corresponding to the moving target detection tasks in the aforementioned embodiment), road drivable area segmentation detection tasks (corresponding to the road segmentation detection tasks in the aforementioned embodiment), and lane line edge detection tasks (corresponding to the lane line detection tasks in the aforementioned embodiment) stored at frame times t-1, t-2 to tn is read from the historical cache Memory, and combined with the posture conversion relationship between each historical frame time (t-1, t-2...tn) and the current frame time t, the expression of each detection result is reconstructed to generate a corresponding high-dimensional reconstructed Embedding vector that is spatially aligned with the current frame time t.

[0186] In some implementations, the frame times t-1, t-2 to tn in the historical cache Memory are as follows: Figure 6 As shown, the FIFO buffer 600 buffers the information of the frames at time t-1, t-2 to tn, wherein the t-1 frame buffer information 601 includes the t-1 frame global information 604, TopK d t-1 frames of 3D object detection task information 605, TopK s t-1 frames of road segmentation detection task information 606, TopK l t-1 frame lane line edge detection task information 607, t-1 frame global information 604 includes t-1 frame pose information 602 and t-1 frame timestamp information 603; tk frame cache information 611 includes tk frame global information 614, TopK d tk frames of 3D object detection task information 615, TopK s tk frames of road segmentation detection task information 616, TopK l tk frame lane edge detection task information 617, tk frame global information 614 includes tk frame pose information 612 and tk frame timestamp information 613. Specifically, taking the tk moment as an example, the information stored for this moment includes: the pose E corresponding to the tk moment of the vehicle relative to the global coordinate system t-k and timestamp T t-k ;TopK d Context Embedding vector corresponding to the 3D object detection task Central Location Target movement speed TopK s The context embedding vector corresponding to the road segmentation detection task Central Location TopK l The context embedding vector corresponding to the lane edge detection task Central Location In general, considering that the dimensions of the embedding vectors of various task contexts are C, the dimensions of the 3D target center position and speed are 3 and 2 respectively, the dimension of the center position of the road and lane line is 2, and the dimensions of the timestamp and the vehicle posture information at the moment are 3×3+3×1+1, the total occupancy of the cache memory is N×(13+K d ×(C+5)+K s ×(C+2)+K l ×(C+2)).

[0187] In some implementations, the pose conversion relationship between each historical frame moment (t-1, t-2, ... tn) and the current frame moment t is combined to reconstruct the information expression of each task. Taking the historical moment tk as an example, a high-dimensional reconstructed Embedding vector corresponding to each detection task at moment tk and aligned with the current frame moment t in space is generated, such as Figure 7 , Figure 8 , Fig. 9 As shown, the operation is as follows:

[0188] First, calculate the time difference between time tk and time t, △T = T t -T t-k , and the vehicle posture transformation relationship between time tk and time t And according to The center positions of the 3D target detection tasks, road segmentation detection tasks, and lane line detection tasks stored at time tk are transformed and aligned to the current vehicle coordinate system. Since the dimensions of the center positions of the roads and lane lines are only 2, the height value of the ground in the vehicle coordinate system needs to be supplemented in the height Z-axis direction to expand it to 3 dimensions, and then calculated according to the conversion formula

[0189] Then, using the vehicle posture transformation relationship Time difference △T and target movement speed V (t-k) , according to the motion-aware regularization (MLN) method, the MLN corresponding coefficients (γ, β) are calculated as follows:

[0190]

[0191] Among them, ξ γ and βRepresents two independent linear transformation layers, the network weights are obtained by learning, and the target motion speed is for 3D targets For road and lane line target motion speed V (t-k) =0, that is, the road passable area target and the lane line target have the same coefficient (γ, β), while for the 3D moving target, due to its different speeds, the obtained coefficients (γ, β) are also different.

[0192] Finally, combined with the MLN coefficients (γ, β) obtained above, the task information cached at time tk is aligned with the Embedding at time t for reconstruction. The calculation method is as follows:

[0193]

[0194] Among them, MLP represents multi-layer perceptron network, LN represents layer regularization. It should be pointed out that since the physical object corresponding to the road segmentation detection is the center position of each square patch on the ground plane of the BEV space, it is directly used right After conversion, the Generally, it is difficult for the square patches of the BEV space in the vehicle coordinate system at the current time t to coincide with each other. Therefore, the nearest patch principle can be adopted, that is, The nearest patch center position is taken as The value is then brought into the Embedding reconstruction calculation.

[0195] Step S403: construct and initialize the current frame Query;

[0196] In some embodiments, an Embedding vector is reconstructed based on the previous frame at time t-1 that is closest to the current frame at time t, and is propagated to initialize the Query information of the current frame. At the same time, combined with the random initialization of part of the current frame's spatial position, the initial Query construction including dynamic and static tasks such as 3D targets, road passable areas, and lane lines is completed together. The feature vector content includes the context Embeddings and position Embeddings corresponding to all queries.

[0197] In some implementations, initializing the current frame Query specifically includes constructing queries corresponding to dynamic and static tasks such as 3D target detection, road traversable area segmentation, lane line edge detection, etc., and the meaning of the physical object is as follows: Fig.10As shown, the BEV space 1001 includes the vehicle center 1002, the 3D target detection task Query309, the road segmentation detection task Query310 and the lane line detection task Query311. The 3D target detection task Query309 corresponds to the initial position and context features of each candidate 3D target in the BEV space; the road segmentation detection task Query310 corresponds to the BEV ground plane S H ×S W The context features corresponding to rectangular patches of equal size are fixed at the center points of each patch; the lane line detection task Query311 corresponds to the initial geometric center position and context features of each lane line or curb.

[0198] In some implementations, the query initialization information of each dynamic and static task of the current frame includes two parts. The first part is to propagate the Embedding vector Propagation reconstructed at time t-1 of the previous frame to initialize the query information of the current frame, that is: The second part is to randomly initialize a batch of queries at the current frame time t, that is

[0199] In some embodiments, the first part is to reconstruct the Embedding vector Propagation at the previous frame t-1 to initialize the TopK of the 3D target task in the current frame Query. d indivual and TopK of lane line tasks l indivual They are directly concatenated with the corresponding task query in the second part, and the TopK of the road task s indivual In addition to being used in conjunction with the corresponding task query in the second part, it is also necessary to confirm that the Embeddings reconstructed at time t-1 correspond to the 2D ground plane S in the BEV space in the current frame vehicle coordinate system. H ×S W The position of the rectangular patch, that is, S H ×S W TopK in road task query s Initialized by Propagation.

[0200] In some embodiments, the specific operation mode of the second part random initialization is: the center position of each 3D target task Query is initialized using the uniformly distributed coordinates in the 3D space of the BEV space under the vehicle coordinate system; the center position of each road task Query is initialized using S H ×S WThe remaining S after the previous frame propagation in the rectangular patch H ×S W TopK s The center position coordinates of each facet; the center position of each lane line along the Query is the uniformly distributed coordinates of the position on the 2D ground plane of the BEV space in the vehicle coordinate system; the initial value of the context Embedding vector of each task is set to 0 vector; in order to maintain feature consistency, the query position coordinate vectors of each task obtained above and the context initialization Embedding vector are substituted into formulas (3) to (6) to obtain the final second part of the initialization Embedding, that is: In formula (3), we can set Substitute and calculate the MLN coefficients (γ, β).

[0201] Step S404: HAS-Propagation Transformer calculation;

[0202] like Fig.11 As shown, the information obtained according to steps S401, S402, and S403 is used as the input of the HAS-PropagationTransformer network module, which contains several layers of hybrid attention mechanism 1102 (HybridAttention) and surround view attention mechanism 1101 (AS Attention) in series. The corresponding specific inputs are: according to the historical frame (t-1, t-2...tn) obtained in step S402, the dynamic and static task Embedding and the query information initialized at the current frame t obtained in step S403, the two information are spliced ​​to form the input Key vector and Value vector of HybridAttention, and the implicit feature association relationship between dynamic targets and static road structures, between road passable areas and roadside lanes, and between current states and historical states is established through the Attention mechanism; then, the output of Hybrid Attention is used as the query input of the subsequent AS Attention, and according to the 2D encoding features of the surround view and surround view images obtained in step S401 and 3D position encoding features The input Key vector and Value vector of ASAttention are formed, and then the feature association relationship between each dynamic and static task and the observation content of the surrounding and surround image at the current frame t is established through the Attention mechanism. Finally, after several layers of Hybrid Attention module and AS Attention module calculation, the current frame Query completes the information association and update expression between each dynamic and static task and each historical moment.

[0203] In some implementations, the Hybrid Attention module is mainly used to construct the association between each task at the current moment and the corresponding historical tasks, and between dynamic and static tasks, while the AS Attention module is mainly used to construct the association between dynamic and static tasks and the current panoramic and surround observations. Specifically, both Hybrid Attention and AS Attention use the multi-head Attention calculation method in the standard Transformer, and the formula is as follows:

[0204]

[0205] Where Q, K, and V represent the Attention module input Query (corresponding to the query matrix in the above embodiment), Key (corresponding to the key matrix in the above embodiment), and Value (corresponding to the value matrix in the above embodiment), respectively. k represents the dimension of K vector, W i Q , W i K , W i V , W O are the weight coefficients of multi-head Attention respectively.

[0206] In some embodiments, the Hybrid Attention module input and output are as follows: Fig.11 As shown, the specific operations are:

[0207] First, set the Query to reconstruct the Embedding vector Propagation from the previous frame t-1 to the current frame t and a batch of random initializations at the current frame t The joint splicing composition is denoted as

[0208] Then, set the Key and Value to be exactly the same, and align the high-dimensional reconstructed Embedding vector of the current frame at time t with the remaining time (t-2...tn) after the historical cache excluding time t-1 The current initialization with the aforementioned build The combined composition is and

[0209] Finally, substitute the query, key, and value constructed above into the multi-head attention calculation formula (7) to obtain the output of the Hybrid Attention module. Used as input for subsequent AS Attention modules.

[0210] In some embodiments, the AS Attention module input and output are as follows: Fig.11 As shown, the specific operations are:

[0211] First, Query is set as the output of the Hybrid Attention module, that is,

[0212] Then, Value is set to all the 2D coding features of the surrounding images in step S401 and surround image 2D encoding features Splicing combination, that is

[0213] Again, Key is set to all the 2D encoding features of the surrounding images in step 301 and 3D position features Add and surround image 2D encoding features and 3D position features Add and combine them together to get

[0214]

[0215] Finally, substitute the query, key, and value constructed above into the multi-head attention calculation formula (7) to obtain the output of the ASAttention module If the layer is not the last layer, Used as the input of the next layer of HAS-PropagationTransformer network module, if this layer is the last layer, It is used for 3D target detection, road drivable area segmentation and lane line edge detection and decoding.

[0216] Step S405: dynamic and static multi-task decoding;

[0217] In some implementations, the 3D target query, the road drivable area query, and the lane line edge query output by the HAS-Propagation Transformer network module obtained in step S404 are respectively connected to the decoding head network corresponding to each task to complete the final dynamic and static target and environment perception multi-task output; then, the decoding confidence of each task query is sorted in descending order from high to low, and the top K d 3D targets, TopK s Road segmentation and TopK lThe corresponding decoding information of each lane line is queried, and the historical cache Memory is updated using the first-in-first-out (FIFO) principle for use in the next frame processing.

[0218] In some implementations, the 3D target query, the road drivable area query, and the lane line query finally output by the HAS-Propagation Transformer are decoded respectively, wherein: the decoding of the 3D target query can adopt a simple multi-layer perceptron network, the input dimension is C, and the output dimension is aligned with the information such as the target confidence, target category, center coordinates, length, width and height of the rectangular box, target direction and target speed, which is specifically set according to actual needs; the decoding of the road drivable area query requires segmenting the pixel-level position area on the BEV plane. Since each road query corresponds to the BEV plane S H ×S W One of the rectangular patches, each patch corresponds to r h × w Pixel size, so the input dimension is C and the output dimension is r h × w A simple multi-layer perceptron network with 1-dimensional confidence is added. Of course, a simple CNN network can also be used here to calculate r h × w Pixel segmentation is performed; the lane line edge Query also uses a multi-layer perceptron network like the aforementioned method, with an input dimension of C, but its output dimension depends on the vectorized expression of the lane line edge. Typical expressions include MapTR, PETRv2, and even 3D lane line PersFormer (if 3D lane lines are used, the height coordinates of the center point initialized by the aforementioned Query need to be adapted). It can be measured in combination with actual needs and complexity constraints, and no constraints are imposed here.

[0219] In some implementations, the decoding confidence of the queries for the tasks of 3D object detection, road drivable area segmentation, and lane line detection are sorted in descending order from high to low, and the top K queries are selected. d 3D targets, TopK s Road segmentation and TopK l The decoded information corresponding to each lane line is Query, and then stored in the historical cache Memory data information according to each task, and the historical cache Memory is updated using the First In First Out (FIFO) principle for use in the next frame processing.

[0220] In some implementations, the lane line of the current frame stored in the history buffer Memory is along the center position The specific calculation method is as follows: first extract the coordinates of all k key points {p1, p2, ..., p k}, taking the C-dimensional lane line Query as input, and using the multi-layer perceptron network to obtain the k-dimensional weights [w1,w2,…,w k ], and then calculate the weighted geometric center of the lane line according to the following formula:

[0221]

[0222] It should be pointed out that in order to maintain the validity of the weight distribution of the multilayer perceptron network, the coordinates of the k key points {p1, p2, …, p k} will be sorted according to certain rules and logic based on the decoded information in advance.

[0223] Step S406: sample library preparation, loss function design, model training and inference.

[0224] In some embodiments, supervised training is performed on the BEV perception network model described in steps S401 to S405, and 2D visual images of the vehicle-mounted surrounding pinhole camera and the surrounding fisheye camera at each viewing angle corresponding to each moment are collected to achieve sensor timestamp data alignment and pairing; then, for different specific perception tasks, corresponding task data sample annotations are completed to generate the true value required for supervised training of the network model; finally, by constructing a corresponding multi-task loss function and combining an effective strategy to accelerate training convergence, the generalization training of the BEV perception network model is completed to obtain robust optimal network weights, and the optimal network weights obtained through training are provided to the network model for reasoning.

[0225] In some embodiments, when the 2D visual images of the vehicle-mounted peripheral pinhole cameras and the surround fisheye cameras at various viewing angles are aligned and paired with sensor timestamp data, if the exposure frequencies are the same, hardware synchronization trigger alignment can be considered, but if the exposure frequencies are different, the nearest neighbor time pairing method can also be used. Generally, visual sensors have a higher sampling frequency, so the delay error introduced by the nearest neighbor pairing is generally controllable.

[0226] In some embodiments, the total loss function L for model training is composed of the weighted loss of all perceptual task predictions, where the 3D object detection loss L det It can generally be obtained by weighted calculation of classification Focal loss and 3D box regression L1 loss. The road drivable area segmentation loss L segIt can generally be obtained by cross entropy calculation, while the lane line edge detection loss can be obtained by calculating the key points and geometric center position deviations through L1 loss. The random data enhancement methods used in network training to improve generalization include but are not limited to random scale transformation, region cropping, symmetric mirroring, color transformation, grid masking, etc.

[0227] Based on the above embodiments, this embodiment provides a vehicle environment detection device, such as Fig.12 As shown, the vehicle environment detection device 1200 includes: a feature extraction module 1201, which is used to extract features from the surrounding view image and the surround view image of the vehicle respectively, and obtain the surrounding view image feature and the surround view image feature; a first determination module 1202, which is used to determine the first query information based on the task type corresponding to the detection task; a second determination module 1203, which is used to determine the first key information and the first value information based on the surrounding view image feature and the surround view image feature; a processing module 1204, which is used to process the first query information, the first key information and the first value information by using the attention mechanism to obtain the target fusion feature; a third determination module 1205, which is used to detect the surrounding environment of the vehicle based on the target fusion feature, and determine the detection result corresponding to the detection task.

[0228] In some embodiments, the surrounding view image features include two-dimensional surrounding view features and three-dimensional surrounding view position features, and the surrounding view image features include two-dimensional surrounding view features and three-dimensional surrounding view position features; the feature extraction module includes: a first encoding unit, used to perform two-dimensional encoding on the surrounding view image and the surrounding view image, and determine the two-dimensional surrounding view features and the two-dimensional surrounding view features; a second encoding unit, used to perform three-dimensional encoding on the surrounding view image and the surrounding view image, and determine the three-dimensional surrounding view position features and the three-dimensional surrounding view position features.

[0229] In some embodiments, the first determination module includes: a first determination unit, used to determine the second query information, second key information and second value information corresponding to the detection task based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment; a processing unit, used to use the attention mechanism to process the second query information, the second key information and the second value information to obtain the first query information.

[0230] In some embodiments, the first determination unit includes: a first determination subunit, used to reconstruct the detection results corresponding to the detection tasks at each of the historical moments based on the posture conversion relationship between each of the historical moments and the vehicle at the current moment, and determine the reconstruction features corresponding to the detection tasks at each of the historical moments; a second determination subunit, used to determine the initial reconstruction features corresponding to the detection task at the current moment based on the task type corresponding to the detection task and the reconstruction features corresponding to the detection task at the first historical moment; the first historical moment is the previous moment of the current moment; a third determination subunit, used to determine the initial reconstruction features corresponding to the detection task at the current moment as the second query information; a fourth determination subunit, used to determine the second key information and the second value information based on the reconstruction features corresponding to the detection tasks at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment; the second historical moment is a historical moment other than the first historical moment.

[0231] In some embodiments, the first determination subunit is also used to: based on the posture conversion relationship, align the task information corresponding to the detection task at each of the historical moments to the current moment, and determine the updated detection results corresponding to the detection tasks at each of the historical moments; based on the posture conversion relationship, the time difference between each of the historical moments and the current moment, and the movement speed corresponding to the detection task, reconstruct the updated detection results corresponding to the detection tasks at each of the historical moments, and determine the reconstruction features corresponding to the detection tasks at each of the historical moments.

[0232] In some embodiments, the second determination subunit is also used to: determine a first initial reconstruction feature corresponding to the detection task at the current moment based on the reconstruction feature corresponding to the detection task at the first historical moment; randomly generate a second initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task; and concatenate the first initial reconstruction feature and the second initial reconstruction feature to determine the initial reconstruction feature corresponding to the current moment.

[0233] In some embodiments, the second determination subunit is also used to: randomly generate an initial feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task; reconstruct the initial feature corresponding to the detection task to determine a second initial reconstructed feature corresponding to the detection task at the current moment.

[0234] In some embodiments, the second determination subunit is also used for at least one of the following: when the detection task includes a moving target detection task, uniformly sampling the coordinates in the three-dimensional space to determine the initial features corresponding to the moving target detection task; when the detection task includes a road segmentation detection task, based on the coordinates of the center position of each sub-area in the set vehicle passable area, determining the initial features corresponding to the road segmentation detection task; when the detection task includes a lane line detection task, uniformly sampling the coordinates in the two-dimensional space to determine the initial features corresponding to the lane line detection task.

[0235] In some embodiments, the second determination module includes: a second determination unit, used to splice the two-dimensional surrounding vision feature and the two-dimensional surrounding vision feature to determine the first key information corresponding to the detection task; a splicing unit, used to splice the two-dimensional surrounding vision feature, the three-dimensional surrounding vision position feature, the two-dimensional surrounding vision feature and the three-dimensional surrounding vision position feature to determine the first value information corresponding to the detection task.

[0236] In some embodiments, the processing module includes: a generation unit, used to generate an attention matrix by using the first query information as the query matrix of the attention mechanism and the first key information as the key matrix of the attention mechanism; and a third determination unit, used to determine the target fusion feature based on the attention matrix and the first value information.

[0237] In some embodiments, the first determination module includes: a fourth determination unit, used to determine, for each of the detection tasks, the query information corresponding to the detection task based on the task type corresponding to the detection task; and a fifth determination unit, used to splice the query information corresponding to each of the detection tasks to determine the first query information.

[0238] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0239] This embodiment also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements part or all of the steps in the above method when executing the program.

[0240] This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0241] This embodiment further proposes a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0242] This embodiment also proposes a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0243] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0244] It should be noted that the embodiment of the present application provides a hardware entity of a computer device, such as Fig.13 As shown, the hardware entity of the computer device 1300 includes: a processor 1301 generally controls the overall operation of the computer device 1300. A communication interface 1302 can enable the computer device to communicate with other terminals or servers through a network. A memory 1303 is configured to store instructions and applications executable by the processor 1301, and can also cache data to be processed or processed by the processor 1301 and each module in the computer device 1300 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 1301, the communication interface 1302, and the memory 1303 through a bus 1304.

[0245] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0246] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0247] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0248] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the embodiments of the present application may be all integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0249] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0250] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0251] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A vehicle environment detection method, characterized in that: include: Extracting features of the vehicle's surrounding image and surround image respectively to obtain surrounding image features and surround image features; Determining first query information based on a task type corresponding to the detection task; Determining first key information and first value information based on the surrounding view image feature and the all-around view image feature; Using an attention mechanism, the first query information, the first key information, and the first value information are processed to obtain a target fusion feature; Based on the target fusion features, the surrounding environment of the vehicle is detected to determine a detection result corresponding to the detection task.

2. The method according to claim 1, characterized in that The surrounding image features include two-dimensional surrounding features and three-dimensional surrounding position features, and the surround image features include two-dimensional surround features and three-dimensional surround position features; The feature extraction of the surrounding view image and the surround view image of the vehicle is performed respectively to obtain the surrounding view image feature and the surround view image feature, including: Performing two-dimensional coding on the surrounding view image and the all-around view image to determine the two-dimensional surrounding view feature and the two-dimensional all-around view feature; The surrounding view image and the all-around view image are three-dimensionally encoded to determine the three-dimensional surrounding view position feature and the three-dimensional all-around view position feature.

3. The method according to claim 1, characterized in that The determining of the first query information based on the task type corresponding to the detection task includes: Determine second query information, second key information, and second value information corresponding to the detection task based on a task type corresponding to the detection task and a detection result corresponding to the detection task at each historical moment; The second query information, the second key information and the second value information are processed by using the attention mechanism to obtain the first query information.

4. The method according to claim 3, characterized in that The determining, based on the task type corresponding to the detection task and the detection results corresponding to the detection task at each historical moment, the second query information, the second key information, and the second value information corresponding to the detection task includes: Based on the posture conversion relationship between the vehicle at each of the historical moments and the current moment, the detection results corresponding to the detection tasks at each of the historical moments are reconstructed to determine the reconstruction features corresponding to the detection tasks at each of the historical moments; Determine the initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task and the reconstruction feature corresponding to the detection task at the first historical moment; the first historical moment is the moment before the current moment; Determining the initial reconstruction feature corresponding to the detection task at the current moment as the second query information; The second key information and the second value information are determined based on the reconstruction features corresponding to the detection task at each second historical moment and the initial reconstruction features corresponding to the detection task at the current moment; the second historical moment is a historical moment other than the first historical moment.

5. The method according to claim 4, characterized in that The reconstructing the detection results corresponding to the detection tasks at each of the historical moments based on the posture conversion relationship between the vehicle at each of the historical moments and the current moment, and determining the reconstructed features corresponding to the detection tasks at each of the historical moments, includes: Based on the posture conversion relationship, align the detection results corresponding to the detection tasks at each of the historical moments to the current moment, and determine the updated detection results corresponding to the detection tasks at each of the historical moments; Based on the posture conversion relationship, the time difference between each historical moment and the current moment, and the movement speed corresponding to the detection task, the updated detection results corresponding to the detection task at each historical moment are reconstructed to determine the reconstruction features corresponding to the detection task at each historical moment.

6. The method according to claim 4, characterized in that The determining, based on the task type corresponding to the detection task and the reconstruction feature corresponding to the detection task at the first historical moment, the initial reconstruction feature corresponding to the detection task at the current moment comprises: Determining a first initial reconstruction feature corresponding to the detection task at the current moment based on the reconstruction feature corresponding to the detection task at the first historical moment; Based on the task type corresponding to the detection task, randomly generate a second initial reconstruction feature corresponding to the detection task at the current moment; The first initial reconstruction feature and the second initial reconstruction feature are concatenated to determine the initial reconstruction feature corresponding to the current moment.

7. The method according to claim 6, characterized in that The randomly generating the second initial reconstruction feature corresponding to the detection task at the current moment based on the task type corresponding to the detection task includes: Based on the task type corresponding to the detection task, randomly generate an initial feature corresponding to the detection task at the current moment; The initial features corresponding to the detection task are reconstructed to determine a second initial reconstructed feature corresponding to the detection task at a current moment.

8. The method according to claim 7, characterized in that The randomly generating the initial features corresponding to the detection task at the current moment based on the task type corresponding to the detection task includes at least one of the following: In the case where the detection task includes a moving target detection task, uniformly sampling the coordinates of the three-dimensional space to determine the initial features corresponding to the moving target detection task; In the case where the detection task includes a road segmentation detection task, determining initial features corresponding to the road segmentation detection task based on coordinates of center positions of each sub-area in a set vehicle passable area; In the case where the detection task includes a lane line detection task, the coordinates of the two-dimensional space are uniformly sampled to determine the initial features corresponding to the lane line detection task.

9. The method according to claim 2, characterized in that The determining of the first key information and the first value information based on the surrounding view image feature and the all-around view image feature includes: splicing the two-dimensional surrounding vision feature and the two-dimensional surround vision feature to determine the first key information corresponding to the detection task; The two-dimensional surrounding vision feature, the three-dimensional surrounding vision position feature, the two-dimensional surround vision feature and the three-dimensional surround vision position feature are spliced ​​to determine the first value information corresponding to the detection task.

10. The method according to any one of claims 1 to 9, characterized in that The using of the attention mechanism to process the first query information, the first key information and the first value information to obtain a target fusion feature includes: Using the first query information as a query matrix of the attention mechanism and the first key information as a key matrix of the attention mechanism to generate an attention matrix; Based on the attention matrix and the first value information, the target fusion feature is determined.

11. The method according to any one of claims 1 to 9, characterized in that The number of the detection tasks is multiple; and the determining of the first query information based on the task type corresponding to the detection task includes: For each of the detection tasks, based on the task type corresponding to the detection task, determining query information corresponding to the detection task; The query information corresponding to each of the detection tasks is concatenated to determine the first query information.

12. The method according to any one of claims 1 to 9, characterized in that The method further comprises: Extracting features from radar point cloud data of the vehicle to obtain point cloud features; The determining of the first key information and the first value information based on the surrounding view image feature and the all-around view image feature includes: Based on the surrounding view image feature, the all-around view image feature and the point cloud feature, first key information and first value information are determined.

13. A vehicle environment detection device, characterized in that: include: A feature extraction module is used to extract features from the vehicle's surrounding image and surround image to obtain surrounding image features and surround image features; A first determination module, configured to determine first query information based on a task type corresponding to the detection task; A second determination module is used to determine first key information and first value information based on the surrounding view image feature and the all-around view image feature; A processing module, configured to process the first query information, the first key information and the first value information by using an attention mechanism to obtain a target fusion feature; The third determination module is used to detect the surrounding environment of the vehicle based on the target fusion feature and determine the detection result corresponding to the detection task.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 12 are implemented.

15. A vehicle, characterized in that: Comprising the computer device of claim 14.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

17. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method according to any one of claims 1 to 12 are implemented.

Images