A lighthouse-oriented cognitive multi-scene robust visual 3D target detection method

Abstract

This application provides a robust visual 3D object detection method for multiple scenarios based on lighthouse cognition, relating to the field of autonomous driving technology. The method includes: acquiring an RGB image of the target scene; processing the RGB image using a pre-trained backbone network to obtain a first image feature map; processing the first image feature map using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map; multiplying the semantic feature map and the depth distribution map to obtain a second image feature map; processing the second image feature map using voxel pooling to obtain a BEV feature map; and processing the BEV feature map using a detection head to obtain a 3D object detection result. This application improves the accuracy of object detection in multiple scenarios.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a visual 3D object detection method that is robust to multiple scenarios for lighthouse cognition. Background Technology

[0002] Visual 3D object detection plays a crucial role in autonomous driving systems, helping vehicles perceive their surroundings and accurately identify and locate obstacles. However, different weather conditions lead to variations in image contrast and clarity, and different geographical environments result in different distributions of detection data within images. Therefore, a single visual 3D detection model can only handle one scene. Autonomous vehicles need to integrate multiple perception models to achieve comprehensive scene perception capabilities, thereby expanding their autonomous driving perception capabilities. However, this approach requires more software and hardware resources, increasing the complexity and cost of the vehicle system. Furthermore, the parameters and settings of different models may interfere with each other, leading to system instability. Summary of the Invention

[0003] In view of this, this application provides a robust visual 3D object detection method for lighthouse cognition in multiple scenarios to solve the above-mentioned technical problems.

[0004] In a first aspect, embodiments of this application provide a robust visual 3D object detection method for lighthouse cognition across multiple scenarios, including:

[0005] Obtain the RGB image of the target scene;

[0006] The RGB image is processed using a pre-trained backbone network to obtain the first image feature map;

[0007] The first image feature map is processed using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map;

[0008] The semantic feature map and the depth distribution map are multiplied to obtain the second image feature map; the second image feature map is then processed using voxel pooling to obtain the BEV feature map.

[0009] The BEV feature map is processed using a detection head to obtain 3D target detection results.

[0010] Furthermore, the dynamic deep network includes: a scene-aware discriminator, a semantic processing branch, and a deep processing branch;

[0011] The RGB image is processed using a pre-trained backbone network to obtain the first image feature map; including:

[0012] The scene-aware discriminator is used to process the feature map of the first image to obtain the scene probability vector P, and the scene probability vector P is output to the semantic processing branch and the depth processing branch respectively.

[0013] The semantic processing branch is used to process the first image feature map and the scene probability vector P to obtain the semantic feature map;

[0014] The depth processing branch is used to process the first image feature map and the scene probability vector P to obtain a depth distribution map.

[0015] Furthermore, the scene-aware discriminator includes: a global average pooling layer, a 1×1 convolutional kernel, a first sigmoid function, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a second sigmoid function; wherein the global average pooling layer is connected to the 1×1 convolutional kernel and the first fully connected layer respectively; the first sigmoid function is connected to the 1×1 convolutional kernel and the first fully connected layer respectively; the first fully connected layer, the ReLU activation function, the second fully connected layer, and the second sigmoid function are connected sequentially.

[0016] The scene-aware discriminator processes the first image feature map to obtain a scene probability vector P; including:

[0017] The information of each channel in the first image feature map is compressed into a scalar by using a global average pooling layer, resulting in a compressed vector with a length equal to the number of channels.

[0018] A fused feature map is obtained by processing the compressed vector using a 1×1 convolution kernel;

[0019] The first sigmoid function is used to transform the fused feature map into a value between 0 and 1. This value is then multiplied by the compression vector to obtain the compressed vector of the enhanced features.

[0020] The compressed vector of the enhanced features is processed using the first fully connected layer and the ReLU activation function to obtain an activation vector of length k;

[0021] By using a second fully connected layer and a sigmoid function to weight and normalize the activation vector, a scene probability vector P = (π1, π2, ..., π) of length k is obtained. k ), where π i Let be the probability value of the RGB image being the i-th scene, 1≤i≤k.

[0022] Furthermore, the semantic processing branch includes a dynamic convolutional layer; the dynamic convolutional layer includes: k parallel convolutional kernels, weighted processing units, BN layers, and activation layers; wherein the k parallel convolutional kernels include: convolutional kernel conv1, ..., convolutional kernel conv kThe weighted processing unit uses the scene probability vector P to weight the outputs of k parallel convolutional kernels, and outputs a weighted feature map.

[0023]

[0024] Among them, f i For convolution kernels (conv) i The feature map output after processing the first image feature map f.

[0025] Furthermore, the deep processing branch includes four dynamically connected convolutional blocks (DyBlocks); each DyBlock is composed of multiple dynamically convolutional layers stacked together.

[0026] Furthermore, the method also includes:

[0027] Obtain a training set; the training set includes multiple RGB image samples of k scenes, and the labels of the RGB image samples include: image annotation results and k-dimensional label vectors. For the RGB image sample of the j-th scene, the k-dimensional label vector yj = [0, 0, ..., 1, ..., 0], where 1 appears at the j-th position of the label vector;

[0028] The RGB image samples are processed using a backbone network to obtain the first image sample feature map;

[0029] The first image sample feature map is processed using a dynamic deep network to obtain the sample semantic feature map and the sample depth distribution map;

[0030] The sample semantic feature map and the sample depth distribution map are multiplied to obtain the second image sample feature map; the second image sample features are processed using voxel pooling to obtain the sample BEV feature map;

[0031] The BEV feature map of the sample is processed using the detection head to obtain the prediction result;

[0032] Based on the k-bit binary code and label vector output by the scene-aware discriminator in the dynamic deep network, calculate the binary cross-entropy loss function L. sce ;

[0033] Based on the prediction results and image annotation results of the RGB image samples, the first loss function L is calculated. det ;

[0034] Calculate the total loss function value L:

[0035] L = L det +L sce

[0036] The parameters of the backbone network and the dynamic deep network are updated using the total loss function value L.

[0037] Furthermore, based on the k-bit binary code and label vector output by the scene-aware discriminator in the dynamic deep network, the binary cross-entropy loss function L is calculated. sce ;include:

[0038] The output of the second fully connected layer of the scene-aware discriminator is set to k neurons, and the output of each neuron represents a bit of a binary code.

[0039] The output value of each neuron is activated by the second sigmoid function and truncated to between 0 and 1, that is, output values ​​greater than or equal to 0.5 are set to 1 and output values ​​less than 0.5 are set to 0, resulting in a k-bit binary code;

[0040] Binary cross-entropy loss function L sce The calculation formula is:

[0041]

[0042] Where N is the number of RGB image samples, y ij It is the j-th element of the k-dimensional label vector of the i-th RGB image sample. It is the j-th bit of the k-bit binary code predicted from the i-th RGB image sample.

[0043] Secondly, embodiments of this application provide a robust visual 3D target detection device for multiple scenarios related to lighthouse cognition, comprising:

[0044] The acquisition unit is used to acquire the RGB image of the target scene;

[0045] The feature extraction unit is used to process the RGB image using a pre-trained backbone network to obtain the first image feature map;

[0046] The first processing unit is used to process the first image feature map using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map.

[0047] The second processing unit is used to multiply the semantic feature map and the depth distribution map to obtain the second image feature map; and to process the second image feature map using voxel pooling to obtain the BEV feature map.

[0048] The target detection unit is used to process the BEV feature map using the detection head to obtain 3D target detection results.

[0049] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0051] This application improves the accuracy of target detection in multiple scenarios. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of an ODD scene provided in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of a driving scenario provided in an embodiment of this application;

[0055] Figure 3 A schematic diagram illustrating multiple detection models corresponding to multiple scenarios provided in the embodiments of this application;

[0056] Figure 4 A schematic diagram illustrating a detection model corresponding to multiple scenarios provided in the embodiments of this application;

[0057] Figure 5 A schematic diagram of the dynamic detection model provided in the embodiments of this application.

[0058] Figure 6 A schematic diagram illustrating how the single-vision 3D object detection model provided in this application can adapt to multiple scenarios;

[0059] Figure 7 A schematic diagram of the technical route for a robust visual 3D target detection method for lighthouse cognition in multiple scenarios provided in the embodiments of this application;

[0060] Figure 8 A flowchart illustrating a robust visual 3D object detection method for lighthouse cognition across multiple scenarios provided in this application embodiment;

[0061] Figure 9 A structural diagram of a dynamic deep network provided in an embodiment of this application;

[0062] Figure 10 This is a structural diagram of the scene discriminator provided in the embodiments of this application;

[0063] Figure 11 This is a structural diagram of a dynamic convolutional layer provided in an embodiment of this application;

[0064] Figure 12 This is a functional structure diagram of a robust visual 3D target detection device for lighthouse cognition in multiple scenarios provided in an embodiment of this application.

[0065] Figure 13 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0067] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0068] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0069] Roadside sensing devices, also known as roadside beacons, are infrastructure architectures with a "five-knowledge" functional system. These five knowledge functions are: **Knowledge Layer:** The roadside beacon considers static maps of intersections, intersection models, and sensor information. **Cognitive Layer:** The roadside beacon includes modules for dynamic intersection maps and localization, traffic participant detection, and traffic participant intent recognition, enabling it to sense and identify vehicles, pedestrians, and bicycles within the intersection and infer their behavioral intentions. **Predictive Layer:** Modules for traffic scenario prediction, traffic flow anomaly prediction, and traffic violation prediction can predict future traffic conditions, flow anomalies, and potential traffic violations using historical data and real-time sensor information. **Other-Knowledge Layer:** The end-to-end sensing model and sensor calibration modules are responsible for integrating information from different sensors and data sources to construct a comprehensive model for perceiving and understanding the traffic environment. These models are one source of the roadside beacon system's sensing capabilities. **Common-Knowledge Layer:** The adaptive fusion model module fuses and processes information from different functional layers to generate comprehensive sensing information. This is one of the keys to collaborative sensing. Roadside beacon sensing is an important component of the cognitive layer.

[0070] The Operational Design Domain (ODD) refers to the specific operating conditions designed for an autonomous driving system. Simply put, the ODD defines the scenarios in which autonomous driving is possible. These scenarios are typically determined by three dimensions: driving situation, driving environment, and environmental influences. Figure 1 As shown. Drivers need to adapt to different environments and road conditions, such as driving on highways, city roads, mountain roads, and in rainy or snowy weather. Figure 2 As shown, expanding the autonomous driving ODD (Operational Design Domain) will enable vehicles to better adapt to different driving scenarios and operational design domains, thereby improving vehicle safety and stability and reducing the risk of traffic accidents. If more and more ODD scenarios are covered, vehicles will be able to better adapt to future road and environmental conditions, providing a foundation for the continued development of autonomous driving.

[0071] Visual 3D object detection plays a crucial role in autonomous driving systems, helping vehicles perceive their surroundings and accurately identify and locate obstacles. However, different weather conditions can lead to variations in image contrast and clarity, and different geographical environments can result in different distributions of detection data within images. Therefore, a single visual 3D detection model can only handle one scene. Autonomous vehicles need to integrate multiple perception models to achieve comprehensive scene perception capabilities, thereby expanding the scope of autonomous driving perception. Figure 3 As shown. However, this approach requires more software and hardware resources, increasing the complexity and cost of the vehicle system. Parameters and settings between different models may interfere with each other, leading to system instability.

[0072] If a single visual 3D model can be applied to multiple autonomous driving scenarios, such as Figure 4 As shown, this simplifies the system architecture, reduces system costs, and makes model updates and upgrades easier, thus maintaining the system's up-to-dateness and competitiveness. However, the structural design of a single model limits its ability to adapt to different scenarios. For example, if the model has too few parameters, it can only learn specific features and patterns; if the model's structure is too simple, it may not be able to handle complex scenarios. Therefore, adopting a model structure with more parameters and greater flexibility can improve the adaptability of a single model to different scenarios.

[0073] Based on the above analysis, this application proposes a robust visual 3D object detection method for lighthouse cognition across multiple scenarios. Specifically, it features a single model with more learnable parameters and a more flexible structure, which can improve the network's generalization ability and performance. This method can be effectively applied to different driving scenarios, thereby expanding the perception ODD for autonomous driving.

[0074] The technical solution of this application enables the application of a single visual 3D model to multiple autonomous driving scenarios. This advantage is achieved by proposing a visual 3D object detection framework with more learnable parameters and a more flexible structure. The BEVDepth model improves model accuracy by inputting image features into DepthNet to generate contextual features and depth distribution. Replacing DepthNet with Dynamic DepthNet allows for dynamic parameter handling. A detailed framework diagram is shown below. Figure 5 As shown, Dynamic DepthNet dynamically extracts depth cues from different scenes and uses a Scene-aware Discriminator to dynamically generate the weights of the convolutional kernels in Dynamic DepthNet.

[0075] The technical advantages of this application are:

[0076] 1. This paper proposes a concept of parameter dynamization, specifically by constructing a more flexible dynamic model structure with more learnable parameters. Different parameters are used for different scenarios, achieving an effect similar to integrating multiple visual 3D object detection networks, such as... Figure 6 As shown, a single-vision 3D object detection model employing the concept of dynamic parameterization can adapt to multiple scenarios, such as... Figure 7 As shown.

[0077] 2. A visual 3D object detection framework based on dynamic convolution is proposed. By adding a dynamic convolution module to the network structure, the parameters are made dynamic, which improves the generalization of the model and the object detection accuracy in multiple scenes.

[0078] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0079] like Figure 8 As shown, this application provides a robust visual 3D object detection method for lighthouse cognition across multiple scenarios, comprising the following steps:

[0080] Step 101: Obtain the RGB image of the target scene;

[0081] Step 102: Process the RGB image using the pre-trained backbone network to obtain the first image feature map;

[0082] Step 103: Process the first image feature map using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map;

[0083] Step 104: Multiply the semantic feature map and the depth distribution map to obtain the second image feature map; process the second image features using voxel pooling to obtain the BEV feature map;

[0084] Step 105: Process the BEV feature map using the detection head to obtain the 3D target detection result.

[0085] In this embodiment, as Figure 9 As shown, the Dynamic DepthNet includes: a Scene-aware Discriminator, a semantic processing branch, and a depth processing branch; the semantic processing branch includes a dynamic convolutional layer DyConv; the depth processing branch includes four sequentially connected dynamic convolutional blocks DyBlock; the dynamic convolutional block DyBlock is composed of multiple stacked dynamic convolutional layers DyConv.

[0086] As one possible implementation, a pre-trained backbone network is used to process the RGB image to obtain a first image feature map; including:

[0087] The scene-aware discriminator is used to process the feature map of the first image to obtain the scene probability vector P, and the scene probability vector P is output to the semantic processing branch and the depth processing branch respectively.

[0088] The semantic processing branch is used to process the first image feature map and the scene probability vector P to obtain the semantic feature map;

[0089] The depth processing branch is used to process the first image feature map and the scene probability vector P to obtain a depth distribution map.

[0090] like Figure 10As shown, the scene-aware discriminator includes: a global average pooling layer, a 1×1 convolutional kernel, a first sigmoid function, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a second sigmoid function; wherein, the global average pooling layer is connected to both the 1×1 convolutional kernel and the first fully connected layer; the first sigmoid function is connected to both the 1×1 convolutional kernel and the first fully connected layer; the first fully connected layer, the ReLU activation function, the second fully connected layer, and the second sigmoid function are connected sequentially; the scene-aware discriminator can implement compression and activation operations;

[0091] As one possible implementation, the scene-aware discriminator processes the first image feature map to obtain a scene probability vector P; including:

[0092] The information of each channel in the first image feature map is compressed into a scalar by using a global average pooling layer, resulting in a compressed vector with a length equal to the number of channels.

[0093] A fused feature map is obtained by processing the compressed vector using a 1×1 convolution kernel;

[0094] The first sigmoid function is used to transform the fused feature map into a value between 0 and 1. This value is then multiplied by the compression vector to obtain the compressed vector of the enhanced features.

[0095] The compressed vector of the enhanced features is processed using the first fully connected layer and the ReLU activation function to obtain an activation vector of length k;

[0096] By using a second fully connected layer and a sigmoid function to weight and normalize the activation vector, a scene probability vector P = (π1, π2, ..., π) of length k is obtained. k ), where π i Let be the probability value of the RGB image being the i-th scene, 1≤i≤k.

[0097] like Figure 11 As shown, the dynamic convolutional layer includes: k parallel convolutional kernels, weighted units, BN layers, and activation layers; wherein the k parallel convolutional kernels include: convolutional kernel conv1, ..., convolutional kernel conv k The weighting unit uses the scene probability vector P to weight the outputs of k parallel convolutional kernels, outputting a weighted feature map.

[0098]

[0099] Among them, f i For convolution kernels (conv) i The feature map output after processing the first image feature map f.

[0100] Scene-aware Discriminator can adaptively adjust different convolution kernels (conv). i By applying weights, this attention mechanism strengthens or suppresses the features of different regions of the input features by calculating their weights, so that the model can automatically focus on important information, that is, information related to the scene.

[0101] Dynamic DepthNet uses different convolutional kernels to convolve images of different scenes. Different parameters are used for different scenes. The generation of the convolutional kernels is controlled by a Scene-aware Discriminator, and the scene probability vector is used as weights to control the k convolutional kernels. i The generation of .

[0102] The Scene-aware Discriminator needs to generate different probability vectors P (i.e., weights) for different scenes. If incorrect weights are generated, incorrect convolutional kernels will be generated, which is disastrous for the model. To ensure that the Scene-aware Discriminator can effectively distinguish different scenes and generate correct weights, thus improving its robustness, it is then trained.

[0103] Traditional training methods use a given set of k scenes as the training set, representing the label for each scene as a k-dimensional one-hot vector, where the i-th position is 1 and the other positions are 0. For example, for the j-th scene, its label vector is y. j = [0, 0, ..., 1, ..., 0], where 1 appears at the j-th position. In this case, the k-dimensional probability vector P = D(x) generated by the Scene-aware Discriminator is... j )={π1,π2,...,π k}, which represents the model for input x. j The probability of belonging to each category, i.e., π i Indicates the model predicts x j The probability of belonging to the i-th category. We usually choose the category with the highest predicted probability as the model for input x. jThe final prediction result is obtained, and the cross-entropy between P and the label y is calculated as the model's loss function. However, the above method has drawbacks. When k is large enough, the scene-aware discriminator needs to output a vector with a very large dimension. If the number of scene categories reaches 1000, then the dimension of P also needs to reach 1000, which is a huge burden on the model and will greatly increase the computational cost and memory usage. At the same time, the scene-aware discriminator may also suffer from overfitting because it needs to learn a large number of class features, while the differences between each scene category are not very large, making it difficult to accurately distinguish them in a limited number of data samples.

[0104] To address the aforementioned issues, this embodiment proposes a binary labeling method, representing the label of each scene category as a binary number. For a k-dimensional probability vector P, which contains k positions, each position represents one bit of a binary number, thus representing 2... k -1 distinct scene categories. For example, for the 10th scene category, its label can be represented by its binary form, 1010, which can be converted into a vector form y. 10 = [0, 1, 0, 1, 0, ..., 0], where 1 appears in the 2nd and 4th positions (from left to right). Specifically, the output of the final fully connected (FC) discriminator network can be set to k neurons, each neuron's output representing a bit of the binary code. Then, the output value of each neuron is activated by the sigmoid function and truncated to between 0 and 1, i.e., output values ​​greater than or equal to 0.5 are set to 1, and those less than 0.5 are set to 0, ultimately resulting in a k-bit binary code. For the output k-bit binary code, the binary cross-entropy loss function can be used to measure its difference from the true label. This loss function can be calculated separately for each binary code bit, and each binary code bit is independent.

[0105] Based on the above inventive concept, the method of this embodiment further includes:

[0106] Obtain a training set; the training set includes multiple RGB image samples of k scenes, and the labels of the RGB image samples include: image annotation results and k-dimensional label vectors. For the RGB image sample of the j-th scene, the k-dimensional label vector yj = [0, 0, ..., 1, ..., 0], where 1 appears at the j-th position of the label vector;

[0107] The RGB image samples are processed using a backbone network to obtain the first image sample feature map;

[0108] The first image sample feature map is processed using a dynamic deep network to obtain the sample semantic feature map and the sample depth distribution map;

[0109] The sample semantic feature map and the sample depth distribution map are multiplied to obtain the second image sample feature map; the second image sample features are processed using voxel pooling to obtain the sample BEV feature map;

[0110] The BEV feature map of the sample is processed using the detection head to obtain the prediction result;

[0111] Based on the k-bit binary code and label vector output by the scene-aware discriminator in the dynamic deep network, calculate the binary cross-entropy loss function L. sce ;

[0112] Based on the prediction results and image annotation results of the RGB image samples, the first loss function L is calculated. det ;

[0113] Calculate the total loss function value L:

[0114] L = L det +L sce

[0115] The parameters of the backbone network and the dynamic deep network are updated using the total loss function value L.

[0116] In this embodiment, the binary cross-entropy loss function L is calculated based on the k-bit binary code and label vector output by the scene-aware discriminator in the dynamic deep network. sce ;include:

[0117] The output of the second fully connected layer of the scene-aware discriminator is set to k neurons, and the output of each neuron represents a bit of a binary code.

[0118] The output value of each neuron is activated by the second sigmoid function and truncated to between 0 and 1, that is, output values ​​greater than or equal to 0.5 are set to 1 and output values ​​less than 0.5 are set to 0, resulting in a k-bit binary code;

[0119] Binary cross-entropy loss function L sce The calculation formula is:

[0120]

[0121] Where N is the number of RGB image samples, yi j It is the j-th element of the k-dimensional label vector of the i-th RGB image sample. It is the j-th bit of the k-bit binary code predicted from the i-th RGB image sample.

[0122] Based on the above embodiments, this application provides a robust visual 3D target detection device for multiple scenarios related to lighthouse cognition. (See attached document.) Figure 12As shown, the visual 3D target detection device 200 for lighthouse cognition with robustness across multiple scenarios provided in this application embodiment includes at least:

[0123] Acquisition unit 201 is used to acquire the RGB image of the target scene;

[0124] The feature extraction unit 202 is used to process the RGB image using a pre-trained backbone network to obtain a first image feature map;

[0125] The first processing unit 203 is used to process the first image feature map using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map.

[0126] The second processing unit 204 is used to multiply the semantic feature map and the depth distribution map to obtain the second image feature map; and to process the second image features using voxel pooling to obtain the BEV feature map.

[0127] The target detection unit 205 is used to process the BEV feature map using the detection head to obtain 3D target detection results.

[0128] It should be noted that the principle of the visual 3D target detection device 200 for multi-scene robust lighthouse cognition provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the visual 3D target detection device 200 for multi-scene robust lighthouse cognition provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0129] Based on the above embodiments, this application also provides an electronic device, see below. Figure 13 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, it implements the robust visual 3D target detection method for lighthouse cognition in multiple scenarios provided in this application embodiment.

[0130] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0131] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0132] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0133] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 13 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 13 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0134] It should be noted that, Figure 13 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0135] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the robust visual 3D object detection method for lighthouse cognition across multiple scenarios provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, allowing the electronic device 300 to implement the robust visual 3D object detection method for lighthouse cognition across multiple scenarios provided in this application by executing the built-in or installed executable program.

[0136] The robust visual 3D target detection method for lighthouse cognition in multiple scenarios provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the robust visual 3D target detection method for lighthouse cognition in multiple scenarios provided in this application embodiment.

[0137] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0138] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0139] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0140] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A robust visual 3D object detection method for lighthouse cognition across multiple scenarios, characterized in that, include: Obtain the RGB image of the target scene; The RGB image is processed using a pre-trained backbone network to obtain the first image feature map; The first image feature map is processed using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map; The semantic feature map and the depth distribution map are multiplied to obtain the second image feature map; the second image feature map is then processed using voxel pooling to obtain the BEV feature map. The BEV feature map is processed using the detection head to obtain 3D target detection results; The dynamic deep network includes: a scene-aware discriminator, a semantic processing branch, and a deep processing branch; The first image feature map is processed using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map; including: The scene-aware discriminator is used to process the feature map of the first image to obtain a scene probability vector. , scene probability vector The outputs are sent to the semantic processing branch and the deep processing branch, respectively. The semantic processing branch is used to process the first image feature map and scene probability vector. The process is performed to obtain a semantic feature map; The deep processing branch is used to process the first image feature map and scene probability vector. The data is processed to obtain a depth distribution map; The scene-aware discriminator includes: a global average pooling layer, Convolutional kernel, first sigmoid function, first fully connected layer, ReLU activation function, second fully connected layer, and second sigmoid function; wherein, the global average pooling layer is respectively connected to... The convolutional kernel is connected to the first fully connected layer; the first sigmoid function is connected to... The convolutional kernel is connected to the first fully connected layer; the first fully connected layer, the ReLU activation function, the second fully connected layer, and the second sigmoid function are connected sequentially. The semantic processing branch includes a dynamic convolutional layer; the dynamic convolutional layer includes: k parallel convolutional kernels, weighted processing units, BN layers, and activation layers; The deep processing branch includes four dynamically connected convolutional blocks (DyBlocks); each DyBlock is composed of multiple dynamically convolutional layers stacked together.

2. The method according to claim 1, characterized in that, The scene-aware discriminator is used to process the feature map of the first image to obtain a scene probability vector. ;include: The information of each channel in the first image feature map is compressed into a scalar by using a global average pooling layer, resulting in a compressed vector with a length equal to the number of channels. use The convolutional kernel processes the compressed vector to obtain a fused feature map; The first sigmoid function is used to transform the fused feature map into a value between 0 and 1. This value is then multiplied by the compression vector to obtain the compressed vector of the enhanced features. The compressed vector of enhanced features is processed using a first fully connected layer and a ReLU activation function to obtain a vector of length [length missing]. Activation vector; The activation vector is weighted and normalized using a second fully connected layer and a sigmoid function to obtain a vector of length [length missing]. Scene probability vector ,in, For RGB image, the first The probability value of each scenario. .

3. The method according to claim 2, characterized in that, The k parallel convolutional kernels include: convolutional kernels , , The weighted processing unit utilizes the scene probability vector. The outputs of k parallel convolutional kernels are weighted to produce a weighted feature map. : in, convolution kernel For the first image feature map The feature map output after processing.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the training set; the training set includes Multiple RGB image samples for a scene, with labels for the RGB image samples including: image annotation results and... The label vector of dimension , for the dimensional RGB image samples of a scene, dimensional label vector , where 1 appears in the label vector at the . In each position; The RGB image samples are processed using a backbone network to obtain the first image sample feature map; The first image sample feature map is processed using a dynamic deep network to obtain the sample semantic feature map and the sample depth distribution map; The sample semantic feature map and the sample depth distribution map are multiplied to obtain the second image sample feature map; the second image sample features are processed using voxel pooling to obtain the sample BEV feature map; The BEV feature map of the sample is processed using the detection head to obtain the prediction result; Based on the output of the scene-aware discriminator in the dynamic deep network Given a binary code and a label vector, calculate the binary cross-entropy loss function. ; Based on the prediction results and image annotation results of the RGB image samples, the first loss function is calculated. ; Calculate the total loss function value : Using the total loss function value The parameters of the backbone network and the dynamic deep network are updated.

5. The method according to claim 4, characterized in that, Based on the output of the scene-aware discriminator in the dynamic deep network Given a binary code and a label vector, calculate the binary cross-entropy loss function. ;include: Set the output of the second fully connected layer of the scene-aware discriminator to... There are 10 neurons, and the output of each neuron represents a bit of a binary code. The output value of each neuron is activated by the second sigmoid function and truncated to a value between 0 and 1; that is, output values ​​greater than or equal to 0.5 are set to 1, and those less than 0.5 are set to 0, resulting in a... Bit-bin code; Binary cross-entropy loss function The calculation formula is: in, It is the number of RGB image samples. It is the first RGB image samples The first dimension of the label vector Bit, Through the first Predicted from RGB image samples The first bit of binary code Bit.

6. A robust visual 3D target detection device for lighthouse cognition across multiple scenarios, characterized in that, include: The acquisition unit is used to acquire the RGB image of the target scene; The feature extraction unit is used to process the RGB image using a pre-trained backbone network to obtain the first image feature map; The first processing unit is used to process the first image feature map using a pre-trained dynamic deep network to obtain a semantic feature map and a depth distribution map. The second processing unit is used to multiply the semantic feature map and the depth distribution map to obtain the second image feature map; and to process the second image feature map using voxel pooling to obtain the BEV feature map. The target detection unit is used to process the BEV feature map using the detection head to obtain 3D target detection results; The dynamic deep network includes: a scene-aware discriminator, a semantic processing branch, and a deep processing branch; The first processing unit is specifically used for: The scene-aware discriminator is used to process the feature map of the first image to obtain a scene probability vector. , scene probability vector The outputs are sent to the semantic processing branch and the deep processing branch, respectively. The semantic processing branch is used to process the first image feature map and scene probability vector. The process is performed to obtain a semantic feature map; The deep processing branch is used to process the first image feature map and scene probability vector. The data is processed to obtain a depth distribution map; The scene-aware discriminator includes: a global average pooling layer, Convolutional kernel, first sigmoid function, first fully connected layer, ReLU activation function, second fully connected layer, and second sigmoid function; wherein, the global average pooling layer is respectively connected to... The convolutional kernel is connected to the first fully connected layer; the first sigmoid function is connected to... The convolutional kernel is connected to the first fully connected layer; the first fully connected layer, the ReLU activation function, the second fully connected layer, and the second sigmoid function are connected sequentially. The semantic processing branch includes a dynamic convolutional layer; the dynamic convolutional layer includes: k parallel convolutional kernels, weighted processing units, BN layers, and activation layers; The deep processing branch includes four dynamically connected convolutional blocks (DyBlocks); each DyBlock is composed of multiple dynamically convolutional layers stacked together.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images