A lighthouse cognition-oriented road end visual long-distance target detection method

Abstract

The application provides a lighthouse cognition-oriented road end visual long-distance target detection method, comprising: acquiring an RGB image of a target scene collected by a road end perception subject; processing the RGB image by using a pre-trained image encoder to obtain a multi-scale image feature map; processing the multi-scale image feature map by using a pre-trained height feature extraction network to obtain a height distribution feature map; processing the multi-scale image feature map by using a depth feature extraction network to obtain a depth distribution feature map; processing the multi-scale image feature map, the height distribution feature map and the depth distribution feature map to obtain a height BEV feature map and a depth BEV feature map; performing fusion processing on the height BEV feature map and the depth BEV feature map to obtain a fusion BEV feature map; and processing the fusion BEV feature map by using a detection head to obtain a 3D target detection result. The application improves the detection accuracy and robustness of the road end visual long-distance target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a road end visual long-distance target detection method for lighthouse cognition. BACKGROUND

[0002] Accurate environment perception is a key issue for automatic driving and intelligent transportation. At present, the environment perception method mainly relies on vehicle-mounted cameras. However, due to the limitation of installation height, there are problems such as blind area and unstable long-distance perception in the vehicle end perception system. In recent years, with the rapid development of intelligent infrastructure, it has become possible to use roadside cameras for traffic environment perception. Roadside cameras can obtain a larger perception field of view and realize long-distance observation due to their high installation height compared with vehicle-mounted sensors.

[0003] At present, the following two problems exist in roadside monocular three-dimensional detection:

[0004] Firstly, compared with the relatively fixed installation parameters of vehicle-mounted cameras, each roadside camera has different specifications, such as the up-down pitch angle, left-right roll angle and installation height of the camera, and the difference in external parameters between different intersections poses new challenges to roadside visual three-dimensional detection tasks; secondly, the use of roadside cameras enables the perception system to grasp a wider range of obstacles, resulting in an increase in obstacle density, thereby making the challenges faced by the perception system increasingly complex. SUMMARY

[0005] Therefore, the present application provides a road end visual long-distance target detection method for lighthouse cognition to solve the above technical problems.

[0006] In a first aspect, the present application provides a road end visual long-distance target detection method for lighthouse cognition, comprising:

[0007] obtaining an RGB image of a target scene collected by a road end perception subject;

[0008] processing the RGB image using a pre-trained image encoder to obtain a multi-scale image feature map;

[0009] processing the multi-scale image feature map using a pre-trained height feature extraction network to obtain a height distribution feature map, and processing the multi-scale image feature map using a depth feature extraction network to obtain a depth distribution feature map;

[0010] processing the multi-scale image feature map, the height distribution feature map and the depth distribution feature map to obtain a height BEV feature map and a depth BEV feature map;

[0011] The height BEV feature map and the depth BEV feature map are fused to obtain a fused BEV feature map;

[0012] The fused BEV feature map is processed by using the detection head to obtain a 3D target detection result.

[0013] Further, the image encoder comprises a 2D backbone network and a multi-scale feature fusion network;

[0014] The RGB image is processed by using the pre-trained image encoder to obtain a multi-scale image feature map, comprising:

[0015] The 2D backbone network is pre-trained, and the RGB image I with a size of HxW is processed to obtain an image feature map; H and W represent the height and width, respectively;

[0016] The multi-scale feature fusion network is pre-trained, and the image feature map is processed to obtain a multi-scale image feature map F context with a size of and a channel number of C.

[0017] Further, the height feature extraction network comprises a first residual module, a second residual module, a third residual module and a deformable convolution layer;

[0018] The first residual module, the second residual module and the third residual module have the same structure and are composed of a convolution layer, a BN layer, a ReLU, a convolution layer, a BN layer and an adder, wherein the adder is used to add the input of the convolution layer and the output of the BN layer.

[0019] Further, the multi-scale image feature map, the height distribution feature map and the depth distribution feature map are processed to obtain a height BEV feature map and a depth BEV feature map, comprising:

[0020] The multi-scale image feature map F context and the height distribution feature map H pred are spliced to obtain a fused feature map F fused ; the size of the height distribution feature map H pred is and the channel number is C H ;

[0021] The height distribution feature map H pred and the fused feature map F fused are cross-multiplied to obtain a two-dimensional height fusion feature map

[0022] The two-dimensional height fusion feature map is projected from two dimensions to three dimensions to obtain a three-dimensional space height feature map

[0023] Three-dimensional spatial height feature map Voxel pooling is performed along the height direction to obtain the height BEV feature map.

[0024] For depth distribution feature map D pred and fusion feature map F fused Perform cross product processing to obtain a deep fusion feature map.

[0025] By combining camera intrinsic parameters with predicted depth values, a fused feature map of a two-dimensional plane is obtained. Upgrade to 3D spatial depth feature map

[0026] 3D spatial depth feature map Voxel pooling is performed along the depth direction to obtain the depth BEV feature map.

[0027] Furthermore, the height BEV feature map and the depth BEV feature map are fused to obtain a fused BEV feature map; including:

[0028] Step S1: Utilize cross-attention mechanism to process high BEV feature maps and depth BEV feature map The process is performed to obtain the cross-attention feature map.

[0029]

[0030] Among them, Q p express and Features of the reference pixel p in the stitched bird's-eye view; Q′ p It is a feature map of height BEV in a roadside scene. Or from the deep BEV feature map in the in-vehicle scene Features of the sampled reference pixel p;

[0031]

[0032] Where M is the total number of attention heads, m is the indexed attention head, K is the total number of sampling keys, and k is the indexed sampling key; from feature Q p Predicted attention weights A mpk It is a scalar value in the range [0,1], representing the correlation between the reference pixel p in the m-th attention head and the k-th key, and Δp mpk Indicated based on feature Q pThe offset of the reference pixel point p, V(p+Ap mpk ) represents the height BEV feature or the depth BEV feature located at the position of p+Ap mpk W m and W' m are learnable weights;

[0033] Step S2: performing normalization processing on the cross-attention feature map to obtain a normalized cross-attention feature map

[0034]

[0035] Step S3: processing the normalized cross-attention feature map using a feedforward network to obtain a feature map G; wherein the feedforward network is connected by a multilayer perceptron, a ReLU activation function and a multilayer perceptron;

[0036] Step S4: adding the cross-attention feature map and the feature map G to obtain a feature map F1:

[0037]

[0038] Step S5: updating the depth BEV feature map to the feature map F1, and turning to Step S1; performing Steps S1-S4 to obtain a feature map F2;

[0039] Step S6: updating the depth BEV feature map to the feature map F2, and turning to Step S1; performing Steps S1-S4 to obtain a final fused BEV feature

[0040] Further, the method further includes the step of jointly training the image encoder, the height feature extraction network, the parameters W m and W' m , the feedforward network and the detection head.

[0041] In a second aspect, an embodiment of the present application provides a road end visual long-distance target detection device for lighthouse cognition, comprising:

[0042] An acquisition unit is configured to acquire an RGB image of a target scene collected by a road end perception subject;

[0043] A first processing unit is configured to process the RGB image using a pre-trained image encoder to obtain a multi-scale image feature map;

[0044] The second processing unit is configured to process the multi-scale image feature map by using the pre-trained high-level feature extraction network to obtain a high-level distribution feature map, and process the multi-scale image feature map by using a deep feature extraction network to obtain a deep distribution feature map.

[0045] The BEV processing unit is configured to process the multi-scale image feature map, the high-level distribution feature map and the deep distribution feature map to obtain a high-level BEV feature map and a deep BEV feature map.

[0046] The fusion unit is configured to perform fusion processing on the high-level BEV feature map and the deep BEV feature map to obtain a fusion BEV feature map.

[0047] The target detection unit is configured to process the fusion BEV feature map by using a detection head to obtain a 3D target detection result.

[0048] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the method of the embodiments of the present application when executing the computer program.

[0049] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program, when run by a processor, implements the method of the embodiments of the present application.

[0050] The present application improves the detection accuracy and robustness of the road end visual long-distance target. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the specific embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0052] Figure 1 The technical route diagram of the lighthouse cognition-oriented road end visual long-distance target detection method provided by the embodiments of the present application;

[0053] Figure 2 The flowchart of the lighthouse cognition-oriented road end visual long-distance target detection method provided by the embodiments of the present application;

[0054] Figure 3 The structure diagram of the high-level feature extraction network provided by the embodiments of the present application;

[0055] Figure 4A functional structural diagram of a lighthouse cognition-oriented road end visual long-distance target detection device provided by an embodiment of the present application is provided.

[0056] Figure 5 A functional structural diagram of an electronic device provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art without creative labor based on the embodiments in the present application are within the scope of protection of the present application.

[0059] First, the design idea of the embodiments of the present application is briefly introduced.

[0060] The road end perception device, which can also be referred to as a road end lighthouse, is a facility architecture with a “five-knowledge” function system. The “five-knowledge” functions are: known function layer, the road side lighthouse considers the static map of the intersection, the intersection wholesale model and the sensor information. Cognitive function layer, the road side contains the intersection dynamic map and positioning, traffic participant detection and traffic participant intention recognition module, which can perceive and identify traffic participants such as vehicles, pedestrians and bicycles in the intersection, and infer their behavior intention. The pre-knowledge function layer, the traffic scene prediction, traffic flow anomaly prediction and traffic violation prediction module can predict the future traffic conditions, flow anomalies and possible traffic violations through historical data and real-time sensor information. It knows the function layer, the end-to-end perception model and the sensor calibration module are responsible for integrating information from different sensors and data sources, and constructing models that comprehensively perceive and understand the traffic environment. These models are one of the sources of the perception ability of the road end lighthouse system; public knowledge function layer, the adaptive fusion model module fuses and processes the information from different function layers to generate comprehensive perception information. This is one of the keys to collaborative perception.

[0061] Road end lighthouse perception is an important part of the cognitive function layer. With roadside perception, autonomous vehicles can achieve a global perspective beyond the current field of view, covering blind spots and greatly improving safety. Since the information of the roadside camera can be broadcast to all autonomous vehicles around it, the additional economic cost of roadside equipment is worth it. In addition, roadside perception can also achieve traffic flow control and improve efficiency.

[0062] Currently, monocular three-dimensional detection of road end lighthouse mainly has the following two problems:

[0063] The first is that each roadside camera has different specifications compared to the relatively fixed installation parameters of the vehicle-mounted camera, such as the up-down pitch angle, left-right roll angle, and installation height of the camera. The difference in external parameters between different intersections poses new challenges to roadside visual three-dimensional detection tasks.

[0064] The second is that the use of roadside cameras enables the perception system to grasp a wider range of obstacles, resulting in an increase in obstacle density and making the challenges faced by the perception system increasingly complex.

[0065] To solve the above technical problems, the present application provides a road end visual long-distance target detection method for lighthouse perception. This method uses learnable interactive attention mechanism fusion to realize high-precision and strong robustness detection by encoding depth information that emphasizes close-range perception and height space encoding that is good at long-distance perception. It is a roadside 3D target detection framework suitable for landing deployment, and the technical route is as shown in Figure 1 .

[0066] From the perspective of information fusion, the present application respectively performs adaptive fusion of height information and depth information from the image perspective and the bird's eye view perspective using learnable cross-attention, which can achieve high external parameter robustness and high detection accuracy for images collected by roadside cameras of any installation height.

[0067] After introducing the application scenarios and design ideas of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0068] As shown in Figure 2 , the present application provides a road end visual long-distance target detection method for lighthouse perception, including the following steps:

[0069] Step 101: acquiring an RGB image of a target scene collected by a road end perception subject;

[0070] Step 102: processing the RGB image using a pre-trained image encoder to obtain a multi-scale image feature map;

[0071] The image encoder includes a 2D backbone network and a multi-scale feature fusion network; this step specifically includes:

[0072] The image feature map is obtained by processing an H×W RGB image I using a pre-trained 2D backbone network; H and W represent the height and width, respectively.

[0073] The image feature map is processed using a pre-trained multi-scale feature fusion network to obtain the multi-scale image feature map F. context Size is The number of channels is C.

[0074] Step 103: Use a pre-trained height feature extraction network to process the multi-scale image feature map to obtain a height distribution feature map; use a depth feature extraction network to process the multi-scale image feature map to obtain a depth distribution feature map.

[0075] like Figure 3 As shown, the high-precision feature extraction network includes: a first residual module, a second residual module, a third residual module, and a deformable convolutional layer; the first residual module, the second residual module, and the third residual module have the same structure, each consisting of a convolutional layer, a BN layer, a ReLU layer, a convolutional layer, a BN layer, and an adder connected together, wherein the adder is used to add the input of the convolutional layer to the output of the BN layer.

[0076] In this embodiment, the deep feature extraction network uses an existing network.

[0077] Step 104: Process the multi-scale image feature map, height distribution feature map, and depth distribution feature map to obtain the height BEV feature map and depth BEV feature map;

[0078] In this embodiment, the step specifically includes:

[0079] For multi-scale image feature maps F context and height distribution feature map H pred The fused feature map F is obtained by stitching the images together. fused The height distribution feature map H pred The size is The number of channels is C H ;

[0080] Based on the height distribution feature map, vehicles can be clearly distinguished from the ground. This observation highlights the crucial importance of height information for 3D object detection. However, traditional LSS-based bird's-eye view perception methods inevitably lose height information during the voxel pooling stage. Therefore, a multi-scale image feature map F is first... context and height distribution feature map H predThe fusion is performed, effectively avoiding the problem of losing height information in the voxel pooling process.

[0081] The height distribution feature map H pred and the fusion feature map F fused are cross-multiplied to obtain a two-dimensional height fusion feature map

[0082] The two-dimensional height fusion feature map is projected from two-dimensional to three-dimensional to obtain a three-dimensional spatial height feature map

[0083] The three-dimensional spatial height feature map is voxel-pooled along the height direction to obtain a height BEV feature map

[0084] The depth distribution feature map D pred and the fusion feature map F fused are cross-multiplied to obtain a depth fusion feature map

[0085] The two-dimensional planar fusion feature map is upgraded to a three-dimensional spatial depth feature map using the camera intrinsic parameters combined with the predicted depth value

[0086] The three-dimensional spatial depth feature map is voxel-pooled along the depth direction to obtain a depth BEV feature map

[0087] Step 105: The height BEV feature map and the depth BEV feature map are fused to obtain a fused BEV feature map.

[0088] The bird's eye view features generated by the depth branch have a small distance error when the camera is perpendicular to the ground, which is particularly evident on autonomous vehicles. Therefore, the fusion of bird's eye view features from the height and depth branches is crucial for achieving accurate object detection and providing a unified framework for roadside and in-vehicle platforms.

[0089] When considering the bird's eye view features from the height branch and the depth branch and , it should be noted that there are significant differences in their error distributions. Therefore, simply performing a stacking operation is not sufficient to achieve the height alignment of features corresponding to the same spatial region. To solve this problem, the following fusion process is adopted:

[0090] Step S1: The height BEV feature map and the depth BEV feature map performing processing to obtain a cross-attention feature map

[0091]

[0092] wherein Q p represents and a feature of a reference pixel point p of the spliced bird's eye view; Q' p is a feature of a reference pixel point p sampled from a height BEV feature map or a depth BEV feature map in a vehicle interior scene;

[0093]

[0094] wherein M is a total number of attention heads, m is an index of an attention head, K is a total number of sampling keys, and k is an index of a sampling key; an attention weight A p predicted from the feature Q mpk is a scalar value in the range of [0, 1], representing a correlation between the reference pixel point p and the kth key in the mth attention head, and Δp mpk represents a predicted offset of the reference pixel point p based on the feature Q p , V(p+Δp mpk ) represents a height BEV feature or a depth BEV feature located at the position of p+Δp mpk ; W m and W' m are learnable weights;

[0095] Step S2: performing normalization processing on the cross-attention feature map to obtain a normalized cross-attention feature map

[0096]

[0097] Step S3: performing processing on the normalized cross-attention feature map using a feedforward network to obtain a feature map G; wherein the feedforward network is connected by a multilayer perceptron, a ReLU activation function, and a multilayer perceptron;

[0098] Step S4: adding the cross-attention feature map and the feature map G to obtain a feature map F1:

[0099]

[0100] Step S5: adding the depth BEV feature map The feature map F1 is updated, and the step S1 is entered; the steps S1-S4 are executed to obtain the feature map F2;

[0101] Step S6: the depth BEV feature map The feature map F2 is updated, and the step S1 is entered; the steps S1-S4 are executed to obtain the final fusion BEV feature

[0102] Step 106: the fusion BEV feature map is processed by using the detection head to obtain the 3D target detection result.

[0103] The method further comprises the step of jointly training the image encoder, the height feature extraction network, the parameters W m and W' m , the feedforward network and the detection head.

[0104] Based on the above embodiment, the embodiment of the present application provides a road end visual long-distance target detection device facing lighthouse cognition. As shown in Figure 4 The road end visual long-distance target detection device 200 provided by the embodiment of the present application at least comprises:

[0105] The acquisition unit 201 is configured to acquire an RGB image of a target scene collected by a road end perception subject;

[0106] The first processing unit 202 is configured to process the RGB image by using the pre-trained image encoder to obtain a multi-scale image feature map;

[0107] The second processing unit 203 is configured to process the multi-scale image feature map by using the pre-trained height feature extraction network to obtain a height distribution feature map; and process the multi-scale image feature map by using the depth feature extraction network to obtain a depth distribution feature map;

[0108] The BEV processing unit 204 is configured to process the multi-scale image feature map, the height distribution feature map and the depth distribution feature map to obtain a height BEV feature map and a depth BEV feature map;

[0109] The fusion unit 205 is configured to perform fusion processing on the height BEV feature map and the depth BEV feature map to obtain a fusion BEV feature map;

[0110] The target detection unit 206 is configured to process the fusion BEV feature map by using the detection head to obtain a 3D target detection result.

[0111] It should be noted that the principle of solving the technical problems of the lighthouse cognition-oriented road end visual long-distance target detection device 200 provided by the embodiments of the present application is similar to the method provided by the embodiments of the present application, therefore, the implementation of the lighthouse cognition-oriented road end visual long-distance target detection device 200 provided by the embodiments of the present application can be referred to the implementation of the method provided by the embodiments of the present application, and the repeated parts will not be described here.

[0112] Based on the above embodiments, the embodiments of the present application also provide an electronic device, as shown in Figure 5 The electronic device 300 provided by the embodiments of the present application at least includes a processor 301, a memory 302 and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the lighthouse cognition-oriented road end visual long-distance target detection method provided by the embodiments of the present application when executing the computer program.

[0113] The electronic device 300 provided by the embodiments of the present application can also include a bus 303 connected to different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0114] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0115] The memory 302 can also include a program tool 3025 with a set of (at least one) program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each of these examples or some combination thereof can include the implementation of a network environment.

[0116] The electronic device 300 can also communicate with one or more external devices 304 such as a keyboard, a mouse, a voice input device, a touch input device, a display screen, or a printer. The electronic device 300 can also communicate with one or more devices that enable users to interact with the electronic device 300 (such as a phone, a computer, or the like) and / or one or more devices (such as a router, a modem, or the like) that enable the electronic device 300 to communicate with one or more other electronic devices. Such communication can occur via an Input / Output (I / O) interface 305. Still yet, the electronic device 300 can communicate with one or more networks (such as one or more Local Area Networks (LANs), Wide Area Networks (WANs), and / or the Internet) through a network adapter 306. As Figure 5 illustrated, the network adapter 306 can communicate with the other components of the electronic device 300 through the bus 303. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with the electronic device 300. These components, as well as the electronic device 300 itself, are shown and described in further detail in connection with FIG. 2. Figure 5 It should be appreciated that the software components described herein may be implemented in a variety of manners on a variety of hardware. As one example, the software components can be implemented in or alongside a web browser, a mobile application, or the like. As another example, the software components can be implemented as a standalone application. As another example, the software components can be implemented as a plug-in for a web browser, a mobile application, or the like. As another example, the software components can be implemented as a server-side application, a client-side application, or the like.

[0117] It should be appreciated that the electronic device 300 is merely one example of an electronic device, and should not be taken as limiting the functionality and / or use of embodiments of the application. Figure 5 It should be appreciated that the electronic device 300 is merely one example of an electronic device, and should not be taken as limiting the functionality and / or use of embodiments of the application.

[0118] The embodiments of the application also provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the target detection method of road end vision against camera extrinsic parameter disturbance provided by the embodiments of the application. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the long-distance target detection method of road end vision for lighthouse cognition by executing the built-in or installed executable program.

[0119] The target detection method of road end vision against camera extrinsic parameter disturbance provided by the embodiments of the application can also be implemented as a program product, and the program product includes program codes, and the program codes are used to make the electronic device 300 execute the long-distance target detection method of road end vision for lighthouse cognition provided by the embodiments of the application when the program product is run on the electronic device 300.

[0120] The program product provided by the embodiments of the present application can adopt any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. Specifically, more specific examples (non-exhaustive list) of the readable storage medium include an electrical connection with one or more wires, a portable disc, a hard disk, a RAM, a ROM, an Erasable Programmable Read Only Memory (EPROM), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0121] The program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided by the embodiments of the present application is not limited thereto, and in the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or component.

[0122] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by multiple units.

[0123] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into one step, and / or divided into multiple steps.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A lighthouse cognition-oriented road end visual long-distance target detection method, characterized in that, The method comprises the following steps: obtaining an RGB image of a target scene collected by a road end perception subject; processing the RGB image by using a pre-trained image encoder to obtain a multi-scale image feature map; processing the multi-scale image feature map by using a pre-trained height feature extraction network to obtain a height distribution feature map, and processing the multi-scale image feature map by using a depth feature extraction network to obtain a depth distribution feature map; processing the multi-scale image feature map, the height distribution feature map and the depth distribution feature map to obtain a height BEV feature map and a depth BEV feature map; fusing the height BEV feature map and the depth BEV feature map to obtain a fused BEV feature map; processing the fused BEV feature map by using a detection head to obtain a 3D target detection result; processing the multi-scale image feature map, the height distribution feature map and the depth distribution feature map to obtain a height BEV feature map and a depth BEV feature map; comprising: Multi-scale image feature map And height distribution feature map Perform splicing processing to obtain a fused feature map The size of the height distribution feature map The number of channels is ; And Respectively represent the height and width​ A height distribution feature map and a fusion feature map is subjected to cross multiplication processing to obtain a two-dimensional height fusion feature map ; Highly fused feature map of two dimensions Performing two-dimensional to three-dimensional projection processing to obtain a three-dimensional spatial high feature map ; A three-dimensional spatial height feature map The voxel pooling process is performed along the height direction to obtain a height BEV feature map ; A depth distribution feature map and a fusion feature map is obtained by performing cross multiplication processing ; Fusing a two-dimensional plane feature map using camera intrinsic parameters in combination with predicted depth values Upsampling to a three-dimensional space depth feature map ; Depth feature map for three-dimensional space The voxel pooling process is performed along the depth direction to obtain a depth BEV feature map .

2. The method of claim 1, wherein, the image encoder comprises a 2D backbone network and a multi-scale feature fusion network; processing the RGB image by using a pre-trained image encoder to obtain a multi-scale image feature map; comprising: The pre-trained 2D backbone network is used to process an RGB image with a size of to obtain an image feature map. ​ The pre-trained multi-scale feature fusion network is used for processing image feature maps to obtain multi-scale image feature maps , the size is ; the number of channels is C.

3. The method of claim 2, wherein, the height feature extraction network comprises a first residual module, a second residual module, a third residual module and a deformable convolution layer; The first residual module, the second residual module and the third residual module are the same in structure and are connected by a convolution layer, a BN layer, a ReLU, a convolution layer, a BN layer and an adder, wherein the adder is used for adding the input of the convolution layer and the output of the BN layer.

4. The method of claim 3, wherein, fusing the height BEV feature map and the depth BEV feature map to obtain a fused BEV feature map; comprising: Step S1: processing the high BEV feature map and the depth BEV feature map by using a cross attention mechanism to obtain a cross attention feature map :​​ wherein, represents and a reference pixel of the stitched bird's eye view a feature of is a feature of a reference pixel sampled from or from a depth BEV feature map in a roadside scene in a car interior scene wherein, is the total number of attention heads, is the index of an attention head, is the total number of sampling keys, is the index of a sampling key; from the feature predicted attention weight is a scalar value in the range [0, 1] representing the relevance of the reference pixel and the k-th key in the i-th attention head, and ; ; represents the predicted offset at the reference pixel based on the feature ,​ represents a height BEV feature or a depth BEV feature located at position; and are learnable weights; Step S2: normalizing the cross-attention feature map Step S3: obtaining the normalized cross-attention feature map : Step S3: using a feedforward network on the normalized cross-attention feature map processing to obtain a feature map ; wherein the feedforward network is connected by a multilayer perceptron, an activation function, and a multilayer perceptron. Step S4: add the cross attention feature map and the feature map to obtain a feature map : Step S5: update the depth BEV feature map to a feature map , go to step S1; execute step S1 Step S4, get a feature map ; Step S6: update the depth BEV feature map to a feature map , go to step S1; execute step S1 Step S4, get the final fused BEV feature .

5. The method of claim 4, wherein, The method also comprises a step of jointly training the image encoder, the high-level feature extraction network, the parameters and of the feedforward network and the detection head.

6. A lighthouse-aware road-side visual long-range target detection apparatus, characterized by, comprising: an acquisition unit, configured to acquire an RGB image of a target scene collected by a road end perception subject; a first processing unit, configured to process the RGB image by using a pre-trained image encoder to obtain a multi-scale image feature map; a second processing unit, configured to process the multi-scale image feature map by using a pre-trained height feature extraction network to obtain a height distribution feature map, and process the multi-scale image feature map by using a depth feature extraction network to obtain a depth distribution feature map; a BEV processing unit, configured to process the multi-scale image feature map, the height distribution feature map and the depth distribution feature map to obtain a height BEV feature map and a depth BEV feature map; a fusion unit, configured to fuse the height BEV feature map and the depth BEV feature map to obtain a fused BEV feature map; a target detection unit, configured to process the fused BEV feature map by using a detection head to obtain a 3D target detection result; the BEV processing unit is specifically configured to: Multi-scale image feature map And height distribution feature map Perform splicing processing to obtain a fused feature map The size of the height distribution feature map The number of channels is ; And respectively represent the height and the width.​ A height distribution feature map and a fusion feature map is subjected to cross multiplication processing to obtain a two-dimensional height fusion feature map ; Highly fused feature maps of two dimensions Performing two-dimensional to three-dimensional projection processing to obtain a three-dimensional spatial high feature map ; A three-dimensional spatial height feature map The voxel pooling process is performed along the height direction to obtain a height BEV feature map ; Depth distribution feature map and fusion feature map Cross-multiplication processing is performed to obtain a depth fusion feature map ; Fusing a two-dimensional plane feature map using camera intrinsic parameters in combination with predicted depth values Upsampling to a three-dimensional space depth feature map ; Depth feature map for three-dimensional space The voxel pooling process is performed along the depth direction to obtain a depth BEV feature map .

7. An electronic device, comprising: comprising: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-5.

Images