Beverage perception method, device, equipment and medium of pure visual mode

Abstract

Embodiments of the present application provide a kind of pure vision modal's bev perception method, device, equipment and medium, involve autonomous driving technical field, wherein, the method comprises the following steps: the visual 2d image in the corresponding view angle in data set is cropped, and effective 2d image is generated;Image features in effective 2d image are extracted by perception model, 3d volume pixel space is divided into multiple layers according to height direction, and different layer intervals and different weights are set for different layers, layer interval and weight are used as parameter, image features in effective 2d image are projected to multiple 3d volume pixel space by view conversion, and image features are converted into bev feature;Bev feature is converted into the 3d target frame of the object to be perceived by perception model, and the information of 3d target frame is output.The scheme reduces the input of training data set model, improves the inference speed of perception model, adopts hyperparameter search strategy, and improves the accuracy of perception model.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a purely visual bev perception method, apparatus, device, and medium. Background Technology

[0002] Autonomous vehicles refer to motor vehicles that primarily rely on technologies such as artificial intelligence, visual computing, radar, GPS positioning, and structured road vehicle-to-infrastructure (V2I) communication to enable them to perceive the environment, plan routes, and control autonomously, allowing embedded edge computing terminals to operate them automatically. Since 2020, the industry has begun to popularize the concepts of Level 2 and Level 3 autonomous driving. 2021 was a crucial year for the development of autonomous driving technology, marking the first year that autonomous driving technology entered Level 3. Unlike traditional human-driven vehicles, a core characteristic of autonomous vehicles is the application and dominance of AI technology. The driving process involves a computer continuously collecting various information during the journey, analyzing and learning through machine learning or deep learning, and then controlling the vehicle to achieve a computer-controlled autonomous driving system. The ultimate goal of autonomous vehicles is to ensure safe driving in any road condition and environment, react promptly to extreme situations, and safeguard the safety of the driver, the vehicle, and other road users.

[0003] Currently, autonomous driving is a crucial area for the deployment and implementation of deep learning algorithms. Autonomous driving algorithms are undergoing rapid updates and iterations, with various novel algorithms constantly being proposed in areas such as object detection, lane recognition, object tracking, semantic segmentation, dynamic object trajectory prediction, and multi-task learning. The core framework of autonomous driving algorithms is divided into three parts: environmental perception, decision planning, and control execution. The environmental perception part converts sensor data into machine language representing the vehicle's environment, specifically including object detection, recognition and tracking, environmental modeling, and motion estimation. In recent years, as advanced driver assistance systems have become increasingly accepted by the market, the requirements for autonomous driving perception algorithms have also increased, leading to widespread attention to BEV (Bird Eye View) perception. The BEV perception paradigm primarily provides a unified space, facilitating the fusion of various tasks and sensors. BEV perception systems can convert information from multiple cameras or radars into a bird's-eye view for tasks such as object detection and instance segmentation, more intuitively displaying the size and orientation of objects in the BEV space. However, this places high demands on the computing power of automotive-grade AI chips. Therefore, a large amount of research in academia and industry focuses on how to design embedded-friendly BEV perception algorithm models. Among them, the m2bev algorithm, through its efficient viewpoint switching method, ensures sufficiently high accuracy using only the pure visual modality, and can achieve remarkable inference speed performance on embedded AI chips.

[0004] Currently proposed pure vision BEV algorithms generally have lower object detection accuracy compared to other modalities. In order to ensure real-time performance on embedded AI chips with limited computing power, they adopt network designs with smaller computational loads, which further reduces the detection accuracy of the algorithms. In addition, related algorithms in academia have only been designed and verified on small batches of open source datasets, and the robustness of BEV models in different scenarios cannot be guaranteed. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a purely visual modality-based BEV perception method to solve the technical problems of low detection accuracy and poor robustness in the prior art. The method includes:

[0006] The target region is determined from the visual 2D images under different viewpoints in the dataset. Based on the target region of the visual 2D images under different viewpoints, the visual 2D images under the corresponding viewpoints in the dataset are cropped to generate effective 2D images. The visual 2D images included in the dataset are 2D images without distortion removal collected from multiple viewpoints, and the target region is the region centered on the object to be perceived.

[0007] Image features are extracted from effective 2D images using a perceptual model. The 3D volumetric pixel space is divided into multiple layers according to the height direction, and different layer intervals and weights are set for different layers. The layer intervals and weights are used as parameters, and the image features in the effective 2D images are projected onto the multiple layers of 3D volumetric pixel space through viewpoint transformation, and the image features are converted into BEV features.

[0008] The perceptual model converts BEV features into 3D bounding boxes of the object to be perceived and outputs the information of the 3D bounding boxes.

[0009] This invention also provides a purely visual modality-based BEV perception device to address the technical problems of low detection accuracy and poor robustness in existing technologies. The device includes:

[0010] The 2D image cropping module is used to determine the target region of visual 2D images from different perspectives in the dataset. Based on the target region of the visual 2D images from different perspectives, the module crops the visual 2D images from the corresponding perspectives in the dataset to generate effective 2D images. The dataset includes visual 2D images that are 2D images without distortion correction collected from multiple perspectives, and the target region is the region centered on the object to be perceived.

[0011] The perception module is used to extract image features from the effective 2D image through the perception model, divide the 3D volumetric pixel space into multiple layers according to the height direction, and set different layer intervals and different weights for different layers. Using the layer intervals and weights as parameters, the image features in the effective 2D image are projected onto the multiple layers of 3D volumetric pixel space through viewpoint transformation, and the image features are converted into BEV features.

[0012] The target bounding box information output module is used to convert the BEV features into 3D target bounding boxes of the object to be perceived through the perception model, and output the information of the 3D target bounding boxes.

[0013] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described BEV perception method for any pure visual modality, thereby solving the technical problems of low detection accuracy and low robustness in the prior art.

[0014] This invention also provides a computer-readable storage medium storing a computer program that executes the above-described pure visual modality BEV perception method, thereby solving the technical problems of low detection accuracy and low robustness in the prior art.

[0015] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least:

[0016] Image enhancement is achieved through a cropping method based on visual 2D images. While maintaining the accuracy of the perceptual model, the inference speed of the perceptual model is improved by reducing its input. Simultaneously, the input to the perceptual model is an undistorted 2D image, directly using distorted images to avoid the time-consuming process and information loss associated with distortion correction, further enhancing the inference speed and accuracy of the perceptual model at the board level. Furthermore, a method is proposed that the 3D volumetric pixel space be divided into multiple layers along the height direction, with different layer spacings and weights assigned to different layers. These layer spacings and weights are then used as parameters to project the image features from the effective 2D image onto the multiple layers of the 3D volumetric pixel space through viewpoint transformation, converting the image features into BEV features to improve the accuracy of the perceptual model. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a purely visual modality-based bev perception method provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart of a general visual modality bev perception method;

[0020] Figure 3 This is a flowchart of the model inference module in the general visual modality BEV perception method;

[0021] Figure 4 This is a schematic diagram illustrating the cropping of a visual 2D image according to an embodiment of the present invention, which implements the above-described BEV perception method.

[0022] Figure 5 This is a structural block diagram of a computer device provided in an embodiment of the present invention;

[0023] Figure 6 This is a structural block diagram of a purely visual modality bev sensing device provided in an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0025] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] In this embodiment of the invention, a purely visual modality-based bev perception method is provided, such as... Figure 1 As shown, the method includes:

[0027] Step S101: Determine the target region for the visual 2D images from different perspectives in the dataset. Based on the target region of the visual 2D images from different perspectives, crop the visual 2D images from the corresponding perspectives in the dataset to generate effective 2D images. The visual 2D images included in the dataset are 2D images without distortion removal collected from multiple perspectives, and the target region is the region centered on the object to be perceived.

[0028] Step S102: Extract image features from the effective 2D image through the perception model, divide the 3D volumetric pixel space into multiple layers according to the height direction, and set different layer intervals and different weights for different layers. Using the layer intervals and weights as parameters, project the image features in the effective 2D image onto the multiple layers of 3D volumetric pixel space through viewpoint transformation, and convert the image features into BEV features.

[0029] Step S103: Convert the BEV features into 3D bounding boxes of the object to be perceived using the perception model, and output the information of the 3D bounding boxes.

[0030] like Figure 2 As shown, the general pure visual modality BEV algorithm includes an image streaming module, a preprocessing module, a model inference module (i.e., the perception module mentioned above), a post-processing module, and a visualization module. Figure 3 As shown, the model inference module includes an input module, an image_encode module, a visual transformation module, a bev_encode module, and an output module.

[0031] In practice, to maximize the removal of interference areas, retain effective targets, improve the accuracy of the perception model, and accelerate the inference speed of the perception model, the following steps are taken to determine the target region from visual 2D images at different viewpoints in the dataset, and to crop the visual 2D images at the corresponding viewpoints in the dataset based on the target regions of the visual 2D images at different viewpoints to generate effective 2D images:

[0032] For each viewpoint, an empirical height value is set for each viewpoint, where the empirical height value is less than the total height of the visual 2D image. For each viewpoint, the height of the center point of the object to be perceived in all visual 2D images under each viewpoint in the dataset is calculated to generate a height dataset. The average value of the height dataset for each viewpoint is calculated and used as the center clipping height for each viewpoint. Half of the empirical height value is used as the reference height. The clipping center height plus the reference height is used as the first height, and the clipping center height minus the reference height is used as the second height. The area between the first height and the second height is the target area of ​​the visual 2D image under each viewpoint. The target area of ​​the visual 2D image under each viewpoint is retained, and the image outside the target area is cropped to obtain the effective 2D image under each viewpoint.

[0033] Specifically, in actual captured images, areas such as the sky and car hoods often lack 3D targets, which can interfere with bounding box detection. Most existing algorithms either fail to crop the images or indiscriminately crop from all six cameras (vehicle-mounted cameras installed in multiple directions on the vehicle), resulting in some targets being cropped and reducing the accuracy of the perception model. This invention utilizes a large amount of self-collected datasets from different scenarios to perform region statistics and cropping on targets captured by cameras at different angles, maximizing the removal of interfering areas and retaining valid targets. This improves both the accuracy and speed of the perception model's inference. Figure 4 As shown, the red area represents the retained region. The two images are from different cameras, and it can be seen from the image that the optimal cropping area differs depending on the viewing angle. After statistically analyzing the entire height dataset, the center point of the target object in each viewpoint has an average value in the height direction (center cropping height). During cropping, this average value is used as the center, and cropping is performed vertically by h / 2, where h is a reference height (empirical height value) determined by experience.

[0034] In practice, to maximize the influence of the effective area in the 3D volumetric pixel space and optimize detection accuracy, the following steps are taken to divide the 3D volumetric pixel space into multiple layers along the height direction, and to set different layer intervals and different weights for different layers:

[0035] A set of height intervals is defined, comprising multiple sets of images. Each height value within a given interval is greater than 0 and less than the total height of the visual 2D image. Using this set as input, the optimal input intervals are calculated using hyperparameter search in a deep learning algorithm. The difference between these optimal input intervals is set as the layer spacing in the height direction of the visual 2D image. This difference is negatively correlated with the object density of the corresponding layer in the visual 2D image; higher object density corresponds to a smaller layer spacing. The 3D volumetric pixel space is then divided into multiple layers according to the layer spacing in the height direction of the visual 2D image. The optimal weight for each layer in the 3D volumetric pixel space is calculated using hyperparameter search. The optimal weight for each layer is positively correlated with the object density of the corresponding layer in the visual 2D image; higher object density corresponds to a larger weight.

[0036] Specifically, in the model inference module (i.e., the perception model), existing viewpoint transformation processing typically projects feature maps onto multiple layers of 3D volumetric pixel space. Each layer in the 3D volumetric pixel space has a weight of 1, and the interval between layers is fixed (projecting after uniformly dividing the real 3D space results in many projection points being projected onto areas of the image without target objects). This invention utilizes hyperparameter search (the principle of hyperparameter search is to select a set of near-optimal parameters from a massive number of weight combinations through an optimization algorithm) to assign different intervals to each layer. This allows the 3D volumetric pixel space to cover more effective areas (projecting more points onto areas with concentrated vehicles and pedestrians; therefore, when dividing the projection space, the interval is smaller and the weight is larger in areas with dense vehicles and pedestrians). Then, by assigning different weights to each layer, the effective areas in the 3D volumetric pixel space have a greater influence, achieving the goal of optimizing accuracy.

[0037] Specifically, first, the input set for the hyperparameter search is determined. The input set includes multiple sets of image height interval values ​​(for example, when the total height of the visual 2D image is 2, the input set is [0-0.5, 0.5-1.5, 1.5-2], [0-0.3, 0.3-1.4, 1.4-2], [0-0.4, 0.4-1.2, 1.2-2], [0-0.2, 0.2-1.1, 1.1-2], etc.). Then, the hyperparameter search is used to determine which set in the above input set is the optimal set input interval. Finally, the difference between the optimal set input intervals is set as the layer spacing of the visual 2D image in the height direction.

[0038] In practice, to convert the image features of an undistorted 2D image into BEV features, the following steps are performed: using layer spacing and weights as parameters, the image features in the effective 2D image are projected onto a multi-layer 3D volumetric pixel space through viewpoint transformation, thereby converting the image features into BEV features:

[0039] In 3D space, calculate the points in 3D space corresponding to the projection region of each viewpoint in the pure visual modality; using the layer spacing and weights as parameters, project the points in 3D space onto the effective 2D image, generate an index from image features to BEV features, and convert the image features into BEV features based on the index.

[0040] Specifically, existing model inference modules (i.e., perception models) use training datasets such as Nuscenes (a training dataset primarily collected in Boston and Singapore, using vehicles equipped with one rotating radar, five long-range radar sensors, and six cameras). Nuscenes images are all distortion-corrected. Furthermore, many images captured by in-vehicle cameras currently exhibit some distortion. Therefore, according to existing training models, the preprocessing module needs to correct image distortion during embedded deployment. Distortion correction increases the time consumption of preprocessing operations, hindering real-time inference. Additionally, image loss occurs during distortion correction, reducing model accuracy. The BEV perception method in this embodiment uses distorted images for direct training, ensuring no loss of image information. Distortion correction is unnecessary during training the perception model (i.e., the preprocessing module), improving the perception model's inference speed and accuracy on the board. When converting image features into the BEV feature space, the projection area of ​​each viewpoint is first calculated in 3D space. Then, the points in 3D space are projected onto the distorted image using a function that calculates the projection of 3D points in the world coordinate system onto 2D coordinates in the pixel coordinate system, generating an index from image features to BEV features (i.e., the correspondence between image pixel positions and BEV space positions).

[0041] In practice, to improve the accuracy of orientation regression, the following steps are used to output the 3D target bounding box information to the detection head:

[0042] The perception model outputs the information of the 3D target bounding box to the detection head. The detection head determines the direction of travel of the 3D target bounding box based on the information of the 3D target bounding box. The direction of travel includes forward and backward. The perception model adds the direction of travel to the information of the 3D target bounding box and outputs it.

[0043] Specifically, existing model inference modules (i.e., perception models) typically employ a detection head method. However, the detection head method is not good at regressing the target direction and easily confuses the front and rear of the vehicle. In this embodiment of the invention, a branch specifically for judging the forward and backward directions (determining the direction of travel) is added to the method of the perception model output to the detection head (the output of existing perception models usually includes information such as the center point, length, width, and height of the target object). This effectively solves the problem of poor direction regression.

[0044] In practice, to train the perception model using undistorted images, the following steps are performed:

[0045] In the process of training the perception model, a training dataset is composed of historical visual 2D images collected from multiple perspectives. Based on the occlusion status of the historical object to be perceived in each historical visual 2D image in the training dataset, the image weight of each historical visual 2D image is determined. Based on the image weight of each historical visual 2D image in the training dataset, the loss function is calculated, and the training process of the perception model is controlled according to the magnitude of the loss function.

[0046] In practice, to improve the accuracy of the training dataset by setting occlusion levels, the following steps are proposed to determine the image weight of each historical 2D visual image based on the occlusion status of the historical object to be perceived in each historical 2D visual image in the training dataset:

[0047] Based on the proportion of the historical object to be perceived being occluded in each historical 2D visual image, each historical 2D visual image is divided into occlusion levels. According to the occlusion levels, the weights corresponding to each occlusion level are calculated using hyperparameter search. The weights corresponding to the occlusion levels of each historical 2D visual image are used as the image weights of each historical 2D visual image.

[0048] Specifically, when labeling the training dataset, the occlusion levels of objects are divided into certain levels (e.g., 5 levels: 0%-20%, 20%-40%, 40%-60%, 60%-80%, and 80%-100% (this varies depending on the dataset)). Existing pure visual modality BEV algorithms do not design strategies for the occlusion levels of historical visual 2D images. In this embodiment of the invention, different occlusion levels are set according to the target. When calculating the loss through the loss function, the optimal loss weight for each level is obtained by hyperparameter search. Different weights are assigned to different levels as weights for the images (historical visual 2D images), thereby improving the accuracy of the training dataset (e.g., the accuracy of the Nuscenes dataset is Map and NDS).

[0049] In this embodiment, a computer device is provided, such as... Figure 5 As shown, it includes a memory 501, a processor 502, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described BEV perception method for any pure visual modality.

[0050] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0051] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the above-described pure visual modal bev perception methods.

[0052] Specifically, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0053] Based on the same inventive concept, this invention also provides a purely visual modality BEV perception device, as described in the following embodiments. Since the principle of the purely visual modality BEV perception device in solving the problem is similar to that of the purely visual modality BEV perception method, the implementation of the purely visual modality BEV perception device can refer to the implementation of the purely visual modality BEV perception method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0054] Figure 6 This is a structural block diagram of a purely visual modal bev perception device according to an embodiment of the present invention, such as... Figure 6 As shown, it includes: a 2D image cropping module 601, a perception module 602, and a target box information output module 603. The structure is described below.

[0055] The 2D image cropping module 601 is used to determine the target region of visual 2D images from different perspectives in the dataset, and to crop the visual 2D images from the corresponding perspectives in the dataset according to the target region of the visual 2D images from different perspectives to generate effective 2D images. The dataset includes visual 2D images that are 2D images without distortion removal collected from multiple perspectives, and the target region is the region centered on the object to be perceived.

[0056] The perception module 602 is used to extract image features from the effective 2D image through the perception model, divide the 3D volumetric pixel space into multiple layers according to the height direction, set different layer intervals and different weights for different layers, use the layer intervals and weights as parameters, and project the image features in the effective 2D image onto the multiple layers of 3D volumetric pixel space through viewpoint transformation, and convert the image features into BEV features.

[0057] The target bounding box information output module 603 is used to convert the BEV features into 3D target bounding boxes of the object to be perceived through the perception model, and output the information of the 3D target bounding boxes.

[0058] In one embodiment, the 2D image cropping module includes:

[0059] The empirical height value unit is used to set an empirical height value for each viewpoint, where the empirical height value is less than the total height of the visual 2D image.

[0060] The calculation center clipping height unit is used to calculate the height of the center point of the object to be perceived in all visual 2D images of each viewpoint in the dataset, generate a height dataset, calculate the average value of the height dataset for each viewpoint, and use the average value as the center clipping height for each viewpoint.

[0061] The target region determination unit is used to take half of the empirical height value as the reference height, add the reference height to the crop center height as the first height, subtract the reference height from the crop center height as the second height, and the region between the first height and the second height is the target region of the visual 2D image under each viewpoint.

[0062] The image cropping unit is used to retain the target region of the visual 2D image under each viewpoint and crop the image outside the target region to obtain the effective 2D image under each viewpoint.

[0063] In one embodiment, the sensing module includes:

[0064] The input set determination unit is used to determine the input set, which includes multiple sets of height interval values ​​of images. Each height value within the height interval is greater than 0 and less than the total height value of the visual 2D image.

[0065] The optimal input interval calculation unit is used to take the input set as input and use the hyperparameter search of the ultra-deep learning algorithm to calculate the optimal set input interval. The difference of the optimal set input interval is set as the layer spacing of the visual 2D image in the height direction. The difference of the optimal set input interval is negatively correlated with the object density of the corresponding layer in the visual 2D image. The higher the object density of the corresponding layer in the visual 2D image, the smaller the layer spacing of that layer.

[0066] Spatial segmentation unit, used to divide the 3D volumetric pixel space into multiple layers according to the layer spacing in the height direction of the visual 2D image;

[0067] The optimal weight calculation unit is used to calculate the optimal weight of each layer in the 3D volumetric pixel space using hyperparameter search. The magnitude of the optimal weight of each layer is positively correlated with the density of objects included in the corresponding layer in the visual 2D image. The higher the density of objects included in the corresponding layer in the visual 2D image, the greater the weight of that layer.

[0068] The 3D space point calculation unit is used to calculate the 3D space points corresponding to the projection area of ​​each viewpoint in the pure visual modality.

[0069] The BEV feature transformation unit is used to project points in 3D space onto an effective 2D image using layer spacing and weights as parameters, generate an index from image features to BEV features, and convert the image features into BEV features based on the index.

[0070] In one embodiment, the above-mentioned apparatus further includes:

[0071] The information output module is used to output the travel direction information added to the 3D target bounding box to the detection head.

[0072] In one embodiment, the information output module includes:

[0073] The direction of travel unit is used to output the information of the 3D target box in the perception model to the detection head. The detection head determines the direction of travel of the 3D target box based on the information of the 3D target box. The direction of travel includes forward and backward.

[0074] The information output unit is used to output the direction of travel added to the 3D target bounding box information through the perception model.

[0075] In one embodiment, the dataset training module includes:

[0076] The training dataset generation unit is used to generate a training dataset from historical visual 2D images collected from multiple perspectives during the training of the perception model.

[0077] The image weight determination unit is used to determine the image weight of each historical visual 2D image based on the occlusion status of the historical object to be perceived in each historical visual 2D image in the training dataset.

[0078] The model training unit is used to calculate the loss function based on the image weights of each historical 2D visual image in the training dataset, and to control the training process of the perceptual model based on the magnitude of the loss function.

[0079] In one embodiment, the image weight determination unit is used to classify each historical visual 2D image into occlusion levels according to the proportion of historical objects to be perceived being occluded in each historical visual 2D image; calculate the weight corresponding to each occlusion level using hyperparameter search based on the occlusion level; and use the weight corresponding to the occlusion level of each historical visual 2D image as the image weight of each historical visual 2D image.

[0080] The embodiments of this invention achieve the following technical effects: Direct training on distorted images avoids image loss, improving the inference speed and accuracy of the perception model at the board; image enhancement based on statistical methods reduces the input to the training dataset model while maintaining its accuracy, thereby increasing the inference speed of the perception model; different weights are assigned according to the occlusion level of the object when calculating the loss, thus improving the accuracy of the training dataset model; a hyperparameter search strategy is adopted to find the optimal interval between different layers and the optimal weight for each layer, thereby improving the accuracy of the perception model; and a forward and backward branch is added to the detection head, effectively improving the accuracy of target object orientation regression.

[0081] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A purely visual modality-based bev perception method, characterized in that, include: The target region is determined from the visual 2D images under different viewpoints in the dataset. The visual 2D images under the corresponding viewpoints in the dataset are cropped according to the target region of the visual 2D images under different viewpoints to generate effective 2D images. The visual 2D images included in the dataset are 2D images without distortion removal collected from multiple viewpoints. The target region is the region centered on the object to be perceived. Image features are extracted from the effective 2D image using a perceptual model. The 3D volumetric pixel space is divided into multiple layers according to the height direction, and different layer intervals and weights are set for different layers. The layer intervals and weights are used as parameters, and the image features in the effective 2D image are projected onto the multiple layers of 3D volumetric pixel space through viewpoint transformation, and the image features are converted into BEV features. The 3D volumetric pixel space is divided into multiple layers along the height direction, and different layer spacings and weights are set for different layers, including: Determine an input set, which includes multiple sets of image height interval values, wherein each height value within the height interval is greater than 0 and less than the total height value of the visual 2D image; Using the input set as input, the optimal set input interval is calculated by hyperparameter search of the deep learning algorithm. The difference of the optimal set input interval is set as the layer interval of the visual 2D image in the height direction. The difference of the optimal set input interval is negatively correlated with the object density of the corresponding layer in the visual 2D image. The higher the object density of the corresponding layer in the visual 2D image, the smaller the layer interval of that layer. The 3D volumetric pixel space is divided into multiple layers according to the layer spacing in the height direction of the visual 2D image; The optimal weight of each layer in the 3D volumetric pixel space is calculated using hyperparameter search. The magnitude of the optimal weight of each layer is positively correlated with the density of objects included in the corresponding layer in the visual 2D image. The higher the density of objects included in the corresponding layer in the visual 2D image, the greater the weight of that layer. The bev features are converted into 3D bounding boxes of the object to be perceived using a perception model, and the information of the 3D bounding boxes is output.

2. The purely visual modality-based BEV perception method as described in claim 1, characterized in that, Determine target regions from visual 2D images at different viewpoints in the dataset, and crop the corresponding visual 2D images in the dataset based on the target regions from the visual 2D images at different viewpoints to generate valid 2D images, including: An empirical height value is set for each viewpoint, wherein the empirical height value is less than the total height of the visual 2D image; For each viewpoint, calculate the height of the center point of the object to be perceived in all the visual 2D images in the dataset under each viewpoint, generate a height dataset, calculate the average value of the height dataset for each viewpoint, and use the average value as the center clipping height under each viewpoint; Half of the empirical height value is used as the reference height. The cropping center height plus the reference height is used as the first height. The cropping center height minus the reference height is used as the second height. The area between the first height and the second height is the target area of ​​the visual 2D image under each viewpoint. In the visual 2D image under each viewpoint, the target region of the visual 2D image under that viewpoint is retained, and the image outside the target region is cropped to obtain the effective 2D image under each viewpoint.

3. The purely visual modality-based BEV perception method as described in claim 1, characterized in that, Using the layer spacing and the weights as parameters, the image features in the effective 2D image are projected onto a multi-layer 3D volumetric pixel space through viewpoint transformation, and the image features are converted into BEV features, including: Calculate the points in 3D space corresponding to the projection area of ​​each viewpoint in the pure visual modality; Using the layer spacing and the weights as parameters, the points in the 3D space are projected onto the effective 2D image to generate an index from the image features to the BEV features. Based on the index, the image features are converted into BEV features.

4. The purely visual modality-based bev perception method as described in any one of claims 1 to 3, characterized in that, Also includes: In the perception model, the information of the 3D target box is output to the detection head. The detection head determines the travel direction of the 3D target box based on the information of the 3D target box. The travel direction includes forward and backward. The travel direction is added to the information of the 3D target box through the perception model.

5. The purely visual modality-based bev perception method as described in any one of claims 1 to 3, characterized in that, Also includes: During the training of the perception model, a training dataset is composed of historical visual 2D images collected from multiple perspectives. Based on the occlusion status of the historical object to be perceived in each historical visual 2D image in the training dataset, the image weight of each historical visual 2D image is determined. The loss function is calculated based on the image weights of each historical 2D visual image in the training dataset, and the training process of the perceptual model is controlled according to the magnitude of the loss function.

6. The purely visual modality-based BEV perception method as described in claim 5, characterized in that, Based on the occlusion status of the historical object to be perceived in each historical visual 2D image in the training dataset, the image weights of each historical visual 2D image are determined, including: Each historical visual 2D image is classified into occlusion levels according to the proportion of the historical object to be perceived being occluded in each historical visual 2D image. Based on the occlusion level, the weights corresponding to each occlusion level are calculated using hyperparameter search. The weight corresponding to the occlusion level of each historical visual 2D image is used as the image weight of each historical visual 2D image.

7. A purely visual modality bev sensing device, characterized in that, include: The 2D image cropping module is used to determine the target region of visual 2D images from different perspectives in the dataset, and to crop the visual 2D images from the corresponding perspectives in the dataset according to the target region of the visual 2D images from different perspectives to generate effective 2D images. The dataset includes visual 2D images that are 2D images without distortion correction collected from multiple perspectives, and the target region is the region centered on the object to be perceived. The perception module is used to extract image features from the effective 2D image through a perception model, divide the 3D volumetric pixel space into multiple layers according to the height direction, and set different layer intervals and different weights for different layers. Using the layer intervals and weights as parameters, the image features in the effective 2D image are projected onto the multiple layers of 3D volumetric pixel space through viewpoint transformation, and the image features are converted into BEV features. The perception module includes: The input set determination unit is used to determine the input set, which includes multiple sets of height interval values ​​of images. Each height value within the height interval is greater than 0 and less than the total height value of the visual 2D image. The optimal input interval calculation unit is used to take the input set as input and use the hyperparameter search of the ultra-deep learning algorithm to calculate the optimal set input interval. The difference of the optimal set input interval is set as the layer spacing of the visual 2D image in the height direction. The difference of the optimal set input interval is negatively correlated with the object density of the corresponding layer in the visual 2D image. The higher the object density of the corresponding layer in the visual 2D image, the smaller the layer spacing of that layer. Spatial segmentation unit, used to divide the 3D volumetric pixel space into multiple layers according to the layer spacing in the height direction of the visual 2D image; The optimal weight calculation unit is used to calculate the optimal weight of each layer in the 3D volume pixel space using hyperparameter search. The magnitude of the optimal weight of each layer is positively correlated with the density of objects included in the corresponding layer in the visual 2D image. The higher the density of objects included in the corresponding layer in the visual 2D image, the greater the weight of that layer. The target bounding box information output module is used to convert the BEV features into 3D target bounding boxes of the object to be perceived through the perception model, and output the information of the 3D target bounding boxes.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the bev perception method of the pure visual modality as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the BEV perception method of the pure visual modality as described in any one of claims 1 to 6.

Images