Image and point cloud data fusion method and device based on joint coding

Abstract

The application provides a kind of image and point cloud data fusion method and device based on joint coding, the method comprises: the foreground image and background image are obtained by carrying out semantic segmentation to RGB image;Point cloud data is voxelized;The center point of voxel is converted to camera coordinate system to obtain corresponding pixel;Determine a plurality of foreground voxels according to the pixel corresponding to each voxel and foreground image;The adjacent foreground pixels in the circular area of the foreground pixel corresponding to each foreground voxel are obtained, the features of each adjacent foreground pixel are fused to obtain fusion features, and the fusion features are superimposed to the first feature of foreground pixel to obtain the second feature of foreground pixel;The voxel feature of foreground voxel and the second feature are superimposed to obtain third feature;The third feature of all foreground voxels is processed using self-attention model to obtain the fourth feature of all foreground voxels.The application fuses two modal features using image foreground information, which can enhance the perception ability of roadside light tower.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to an image and point cloud data fusion method and device based on joint coding. BACKGROUND

[0002] A 3D detection algorithm (PointPainting) based on a laser radar point cloud can supplement semantic information of the point cloud by using semantic segmentation in an image, but this method is limited by the result of semantic segmentation, and the information obtained from a 2D plane lacking depth is less, it is difficult to accurately classify and segment pixels of different categories, and there may be a possibility that a 3D object is split and excessive classification information is added. Some methods thicken the laser radar point cloud around the foreground points to facilitate the network to identify the object; however, this method only adds new points and does not use the pseudo-depth feature of foreground segmentation to enhance the semantic information of the original laser point cloud points.

[0003] In the existing method of 3D target detection based on fusion of 4D millimeter wave radar and image features, there is a technical defect that the image feature position is blindly selected, and the image foreground information cannot be effectively used to fuse the features of two modalities. SUMMARY

[0004] Therefore, the present application provides an image and point cloud data fusion method and device based on joint coding to solve the above technical problems.

[0005] In a first aspect, an image and point cloud data fusion method based on joint coding is provided, comprising:

[0006] acquiring point cloud data and an RGB image of the same target scene collected by a roadside beacon tower;

[0007] performing semantic segmentation on the RGB image to obtain a foreground image and a background image;

[0008] performing voxelization processing on the point cloud data to obtain a plurality of voxels each containing at least one point; converting a center point of each voxel to a camera coordinate system to obtain a pixel corresponding to each voxel; and determining a plurality of foreground voxels according to the pixel corresponding to each voxel and the foreground image;

[0009] obtaining adjacent foreground pixels in a circular region of a foreground pixel corresponding to each foreground voxel, fusing features of the adjacent foreground pixels to obtain a fused feature, and superimposing the fused feature on a first feature of the foreground pixel to obtain a second feature of the foreground pixel;

[0010] Superimpose the voxel feature of the foreground voxel and the second feature of the corresponding foreground pixel to obtain a third feature of the foreground voxel; and process the third features of all foreground voxels by using a self-attention model trained in advance to obtain fourth features of all foreground voxels.

[0011] Further, the semantic segmentation model adopts a DeepLabv3 network.

[0012] Performing semantic segmentation on the RGB image to obtain a foreground image and a background image, comprising:

[0013] Processing the RGB image by using a two-dimensional backbone network to obtain an RGB feature map;

[0014] Processing the RGB feature map by using a semantic segmentation model trained in advance to obtain the foreground image and the background image.

[0015] Further, determining a plurality of foreground voxels according to the pixels corresponding to each voxel and the foreground image, comprising:

[0016] Determining whether the pixel corresponding to each voxel belongs to the foreground image, and if yes, the voxel is a foreground voxel.

[0017] Further, obtaining neighboring foreground pixels of the foreground pixel corresponding to each foreground voxel in a circular region with a radius of r of the foreground pixel, comprising:

[0018] Dividing the RGB image into a plurality of grids with a preset size;

[0019] Uniformly sampling or density sampling the pixel points in each grid to obtain a plurality of sampling pixel points;

[0020] Determining a circular region with a radius of r of the foreground pixel corresponding to each foreground voxel;

[0021] Taking the sampling pixel points located in the circular region as the neighboring foreground pixels of the foreground pixel.

[0022] Further, fusing the features of the neighboring foreground pixels to obtain a fused feature, comprising:

[0023] Fusing the features of the neighboring foreground pixels by using maximum pooling, distance weighting or attention weighting to obtain the fused feature.

[0024] Further, processing the third features of all foreground voxels by using a self-attention model trained in advance to obtain fourth features of all foreground voxels, comprising:

[0025] Taking any foreground voxel as a target foreground voxel;

[0026] The pre-trained self-attention model is used to calculate a weight matrix of the third feature of the target foreground voxel relative to the third features of other foreground voxels.

[0027] The weight matrix and the third feature of the target foreground voxel are multiplied to obtain a fourth feature of the target foreground voxel.

[0028] Further, the method further comprises:

[0029] The fourth features of all foreground voxels are processed by using a 3D detector to obtain a 3D target detection result.

[0030] In a second aspect, an image and point cloud data fusion device based on joint coding is provided, comprising:

[0031] An acquisition unit is configured to acquire point cloud data and an RGB image of a same target scene collected by a roadside beacon tower.

[0032] A semantic segmentation unit is configured to perform semantic segmentation on the RGB image to obtain a foreground image and a background image, and determine a plurality of foreground voxels according to pixels corresponding to each voxel and the foreground image.

[0033] A first processing unit is configured to perform voxelization processing on the point cloud data to obtain a plurality of voxels each containing at least one point, convert a center point of each voxel to a camera coordinate system to obtain a pixel corresponding to each voxel, and determine a plurality of foreground voxels according to pixels corresponding to each voxel and the foreground image.

[0034] A second processing unit is configured to acquire adjacent foreground pixels in a circular region of a foreground pixel corresponding to each foreground voxel, fuse features of the adjacent foreground pixels, superimpose the fused features on a first feature of the foreground pixel to obtain a second feature of the foreground pixel.

[0035] A third processing unit is configured to superimpose a voxel feature of the foreground voxel and the second feature of the corresponding foreground pixel to obtain a third feature of the foreground voxel, and process the third features of all foreground voxels by using a pre-trained self-attention model to obtain fourth features of all foreground voxels.

[0036] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiments of the present application when executing the computer program.

[0037] In a fourth aspect, a computer readable storage medium is provided, which stores computer instructions, wherein the computer instructions are executable by a processor to implement the method of the embodiments of the present application.

[0038] The application fuses two modal features by using image foreground information, and can enhance the perception ability of the roadside beacon. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0040] Figure 1 The flowchart of the image and point cloud data fusion method based on joint coding provided by the embodiment of the present application is shown in the figure.

[0041] Figure 2 The functional structure diagram of the image and point cloud data fusion device based on joint coding provided by the embodiment of the present application is shown in the figure.

[0042] Figure 3 The functional structure diagram of the electronic device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0044] Therefore, the detailed description of the embodiments of the present application provided in the drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0045] First, the design idea of the embodiment of the present application is briefly introduced.

[0046] With the increasing popularity of intelligent transportation, roadside light towers can perceive complex traffic scenes such as intersections from a broader perspective and empower autonomous vehicles to improve their perception capabilities. In current target detection algorithms based on three-dimensional point cloud information, some methods voxelize the three-dimensional space to enable three-dimensional discrete convolution for target detection. This method will result in a large amount of calculation and memory cost, and does not fully utilize the sparsity of point sets in three dimensions. Some methods directly operate on points and propagate information through pooling operators or continuous convolution. However, a low-cost and easily accessible data is image data obtained by cameras. However, it is difficult to directly recover accurate depth information from monocular images, so it is often used in combination with other sensors such as laser radar sensors. However, the problem with laser sensors is that they are very expensive and difficult to promote and apply on a large scale. In contrast, the recently emerging 4D millimeter wave radar has the advantages of accurate detection of pitch angle, strong long-distance detection capability, rich features, good adaptability to rain, snow and fog, and low price, and has received widespread attention from academia and industry. However, compared with laser radar, the point cloud generated by 4D millimeter wave radar is more sparse, which makes the traditional 3D target detection method for laser radar cannot be directly applied to 4D millimeter wave radar data. Therefore, it is necessary to enhance the features of 4D millimeter wave radar data, and image and point cloud data feature fusion is one of the methods.

[0047] Among the common multi-modal feature fusion methods, there are three common types: result-level fusion, feature-level fusion and multi-level fusion. Result-level fusion usually uses a 2D target detector to narrow the focus area of the 3D target detector, and obtains a 3D point cloud region based on the detected 2D BoundingBox, and then performs detection. Information aggregation occurs in the result part, and the core idea of result-level fusion is not to use multi-modal data to complement each other, but to reduce the amount of calculation. Feature-level fusion projects point clouds into front view and top view, and projects 3D candidate boxes to each view to extract and fuse regional features, and finally performs bounding box regression. Fusion is performed at the region of interest (ROI) level. Multi-level fusion combines result-level fusion and feature-level fusion, generates 2D BoundingBox using existing 2D target detection, and obtains the corresponding point cloud. Image and point cloud features are extracted using similar ResNet and PointNet networks, and finally 3D BoundingBox is estimated.

[0048] Attention mechanism is first applied in natural language processing, because natural language processing needs to discuss the relationship between words, analyze the words that have the greatest impact on the meaning of the whole sentence. Experiments show that the Transformer network model based on attention mechanism has very good long sequence information modeling ability, and its algorithm performance is better than that of traditional recurrent neural network and long short-term memory network. The self-attention method can adaptively assign the importance of each feature, so as to give more valuable object features by comprehensively considering the channel weight information, which can help the sparse point cloud features in three-dimensional target detection to better extract useful information.

[0049] In order to overcome the technical defects that the existing 3D target detection method based on fusion of 4D millimeter wave radar and image features blindly selects image feature position and cannot effectively utilize image foreground information to fuse two modal features, the application provides an image and point cloud data fusion method based on joint coding. The method supplements the lack of 4D millimeter wave radar data sparsity and small amount of information by fusing image features, enhances the fusion features through attention mechanism, improves the detection performance of the subsequent 3D target detector, and enhances the perception ability of the roadside light tower.

[0050] After introducing the application scenarios and design ideas of the embodiments of the application, the technical solutions provided by the embodiments of the application are described below.

[0051] As shown in Figure 1 The application provides an image and point cloud data fusion method based on joint coding, which comprises:

[0052] Step 101: acquiring point cloud data and RGB images of the same target scene collected by a roadside light tower;

[0053] Among them, the point cloud data is collected by a 4D millimeter wave radar, and the RGB image is obtained by a camera. Data enhancement methods are used to enhance the image and point cloud data, and the enhancement methods include but are not limited to rotation, inversion, scaling, splicing, etc. Data preprocessing is performed to adapt to the model input.

[0054] Step 102: performing semantic segmentation on the RGB image to obtain a foreground image and a background image;

[0055] Among them, the semantic segmentation model adopts a DeepLabv3 network;

[0056] The semantic segmentation on the RGB image to obtain a foreground image and a background image comprises:

[0057] The RGB image is processed by a two-dimensional backbone network to obtain an RGB feature map;

[0058] The RGB feature map is processed by using a pre-trained semantic segmentation model to obtain a foreground image and a background image.

[0059] Step 103: voxelizing the point cloud data to obtain a plurality of voxels each containing at least one point; converting the center point of each voxel to a camera coordinate system to obtain a pixel corresponding to each voxel; determining a plurality of foreground voxels according to the pixel corresponding to each voxel and the foreground image;

[0060] Specifically, determining a plurality of foreground voxels according to the pixel corresponding to each voxel and the foreground image comprises:

[0061] If the pixel corresponding to each voxel belongs to the foreground image, the voxel is a foreground voxel.

[0062] Step 104: obtaining neighboring foreground pixels in a circular region of a foreground pixel corresponding to each foreground voxel, fusing features of the neighboring foreground pixels to obtain a fused feature, and superimposing the fused feature on a first feature of the foreground pixel to obtain a second feature of the foreground pixel;

[0063] The step of obtaining neighboring foreground pixels in a circular region of a foreground pixel corresponding to each foreground voxel comprises:

[0064] The RGB image is divided into a plurality of grids of a preset size; for example, the preset size is 64*64;

[0065] Uniform sampling or density sampling is used to sample a preset number of pixel points in each grid to obtain a plurality of sampled pixel points;

[0066] The radius of the circular region of the foreground pixel corresponding to each foreground voxel is determined as r;

[0067] The sampled pixel points located in the circular region are taken as the neighboring foreground pixels of the foreground pixel.

[0068] Specifically, fusing features of the neighboring foreground pixels to obtain a fused feature comprises:

[0069] The features of the neighboring foreground pixels are fused by using maximum pooling, distance weighting or attention weighting to obtain a fused feature.

[0070] Step 105: superimposing the voxel feature of the foreground voxel and the second feature of the corresponding foreground pixel to obtain a third feature of the foreground voxel; processing the third features of all foreground voxels by using a pre-trained self-attention model to obtain fourth features of all foreground voxels.

[0071] For example, the voxel feature can be obtained by inputting the point cloud data into a 3D encoder network.

[0072] The third feature of all foreground voxels is processed by using the self-attention model trained in advance to obtain the fourth feature of all foreground voxels, including:

[0073] Taking any foreground voxel as a target foreground voxel;

[0074] The weight matrix of the third feature of the target foreground voxel relative to the third feature of other foreground voxels is calculated by using the self-attention model trained in advance;

[0075] The fourth feature of the target foreground voxel is obtained by multiplying the weight matrix and the third feature of the target foreground voxel.

[0076] The weight matrix of the third feature of the target foreground voxel relative to the third feature of other foreground voxels is calculated by using the self-attention model trained in advance, including:

[0077] First, the Q, K and V matrices of each foreground voxel are obtained through Wq, Wk and Wv, and then the weight matrix of the importance of the target foreground voxel relative to other foreground voxels is obtained by performing matrix multiplication on the Q matrix of the target foreground voxel and the K matrix of all other foreground voxels.

[0078] As can be seen, through the self-attention model, the third feature of each target foreground voxel aggregates the third features of all other foreground voxels.

[0079] Further, the method further includes:

[0080] The fourth feature of all foreground voxels is processed by using the 3D detector to obtain a 3D target detection result.

[0081] Illustratively, the 3D detector includes PointPillar, PV-RCNN, etc.

[0082] Based on the above embodiments, the embodiment of the application provides an image and point cloud data fusion device based on joint coding, referring to Figure 2 As shown in the figure, the image and point cloud data fusion device based on joint coding 200 provided by the embodiment of the application at least includes:

[0083] The acquisition unit 201 is configured to acquire point cloud data and RGB images of the same target scene collected by a roadside beacon tower;

[0084] The semantic segmentation unit 202 is configured to perform semantic segmentation on the RGB images to obtain foreground images and background images;

[0085] The first processing unit 203 is configured to perform voxelization processing on the point cloud data to obtain a plurality of voxels each containing at least one point; convert a center point of each voxel to a camera coordinate system to obtain a pixel corresponding to each voxel; and determine a plurality of foreground voxels according to the pixel corresponding to each voxel and the foreground image.

[0086] The second processing unit 204 is configured to obtain adjacent foreground pixels in a circular region of a foreground pixel corresponding to each foreground voxel, fuse features of the adjacent foreground pixels to obtain fused features, and superimpose the fused features on first features of the foreground pixel to obtain second features of the foreground pixel.

[0087] The third processing unit 205 is configured to superimpose voxel features of the foreground voxel and the second features of the foreground pixel corresponding to the foreground voxel to obtain third features of the foreground voxel, and process the third features of all foreground voxels by using a self-attention model trained in advance to obtain fourth features of all foreground voxels.

[0088] It should be noted that the principle of solving the technical problem of the image and point cloud data fusion device 200 based on joint coding provided in the embodiments of the present application is similar to the method provided in the embodiments of the present application, and therefore the implementation of the image and point cloud data fusion device 200 based on joint coding provided in the embodiments of the present application can be referred to the implementation of the method provided in the embodiments of the present application, and the repeated parts will not be described herein.

[0089] Based on the above embodiments, the embodiments of the present application further provide an electronic device, as shown in Figure 3 The electronic device 300 provided in the embodiments of the present application at least includes a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the image and point cloud data fusion method based on joint coding provided in the embodiments of the present application when executing the computer program.

[0090] The electronic device 300 provided in the embodiments of the present application can further include a bus 303 connected to different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0091] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0092] The memory 302 can also include a set of one or more program modules 3024, including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which can include an implementation of a network environment, alone or in combination with another of these examples.

[0093] The electronic device 300 can also communicate with one or more external devices 304 such as a keyboard or a pointing device, for example, by the I / O interface 305. Additionally, the electronic device 300 can communicate with one or more devices that enable user interaction with the electronic device 300 (such as a telephone, for example) and / or one or more devices that enable the electronic device 300 to communicate with one or more other electronic devices 300 (such as a router, for example). Such communication can be enabled by the I / O interface 305. Further, the electronic device 300 can communicate with one or more networks, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, for example, through the network adapter 306. As Figure 3 illustrated, the network adapter 306 is in communication with the other modules of the electronic device 300 through the bus 303. As will be appreciated, although not explicitly illustrated, other mobile computing devices, such as laptops, handheld computers, netbooks, netpads, etc., can utilize an electronic device 300' and the system 300 in a manner similar to that described in connection with the electronic device 300. Figure 3 It should be understood that, although not explicitly illustrated herein, other hardware and / or software modules that can be employed in conjunction with the electronic device 300 can also be utilized. For example, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc., can be utilized.

[0094] It should be understood that the electronic device 300 as illustrated is merely one example and should not be taken to limit the scope of functionality or use of embodiments of the present application. Figure 3 It should be understood that the electronic device 300 as illustrated is merely one example and should not be taken to limit the scope of functionality or use of embodiments of the present application.

[0095] The embodiments of the present application also provide a computer readable storage medium, which stores computer instructions, and the computer instructions are executed by a processor to implement the method provided by the embodiments of the present application. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the joint coding based image and point cloud data fusion method provided by the embodiments of the present application by executing the built-in or installed executable program.

[0096] The method for fusing image and point cloud data based on joint coding provided in the embodiments of the present application can also be implemented as a program product. The program product includes program codes. When the program product can run on the electronic device 300, the program codes are used to make the electronic device 300 execute the method for fusing image and point cloud data based on joint coding provided in the embodiments of the present application.

[0097] The program product provided in the embodiments of the present application can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. Specifically, more specific examples (non-exhaustive list) of the readable storage medium include an electrical connection with one or more wires, a portable disc, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0098] The program product provided in the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided in the embodiments of the present application is not limited to this. In the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or component.

[0099] It should be noted that although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0100] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into one step, and / or divided into multiple steps.

[0101] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the examples, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present application can be made without departing from the spirit and scope of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A method for fusing image and point cloud data based on joint coding, characterized in that, include: Acquire point cloud data and RGB images of the same target scene collected by roadside lighthouses; Semantic segmentation is performed on the RGB image to obtain a foreground image and a background image; Voxelization is performed on the point cloud data to obtain multiple voxels containing at least one point; the center point of each voxel is transformed to the camera coordinate system to obtain the pixel corresponding to each voxel; Multiple foreground voxels are determined based on the pixels corresponding to each voxel and the foreground image; Obtain the neighboring foreground pixels within the circular region corresponding to each foreground voxel, fuse the features of each neighboring foreground pixel to obtain the fused feature, and superimpose the fused feature onto the first feature of the foreground pixel to obtain the second feature of the foreground pixel. The voxel features of the foreground voxel and the second features of the corresponding foreground pixels are superimposed to obtain the third features of the foreground voxel. The third feature of all foreground voxels is processed using a pre-trained self-attention model to obtain the fourth feature of all foreground voxels.

2. The method according to claim 1, characterized in that, Semantic segmentation of the RGB image to obtain a foreground image and a background image includes: RGB images are processed using a two-dimensional backbone network to obtain RGB feature maps; The RGB feature map is processed using a pre-trained semantic segmentation model to obtain foreground and background images; the semantic segmentation model uses the DeepLabv3 network.

3. The method according to claim 1, characterized in that, Multiple foreground voxels are determined based on the pixels corresponding to each voxel and the foreground image, including: Determine whether the pixel corresponding to each voxel belongs to the foreground image. If it does, then the voxel is a foreground voxel.

4. The method according to claim 1, characterized in that, Obtain the neighboring foreground pixels within the circular region corresponding to each foreground voxel, including: Divide the RGB image into multiple grids of a preset size; By using uniform sampling or density sampling, a preset number of pixels are sampled in each grid to obtain multiple sampled pixels; Determine a circular region with radius r corresponding to the foreground pixel of each foreground voxel; The sampled pixels located within the circular area are considered as the neighboring foreground pixels of the foreground pixel.

5. The method according to claim 1, characterized in that, The features of each neighboring foreground pixel are fused to obtain fused features, including: The features of each neighboring foreground pixel are fused using max pooling, distance weighting, or attention weighting to obtain fused features.

6. The method according to claim 1, characterized in that, The third feature of all foreground voxels is processed using a pre-trained self-attention model to obtain the fourth feature of all foreground voxels, including: Use any foreground voxel as the target foreground voxel; The weight matrix of the third feature of the target foreground voxel relative to the third features of other foreground voxels is calculated using a pre-trained self-attention model. The fourth feature of the target foreground voxel is obtained by multiplying the weight matrix and the third feature of the target foreground voxel.

7. The method according to claim 1, characterized in that, The method further includes: The fourth feature of all foreground voxels is processed using a 3D detector to obtain 3D target detection results.

8. An image and point cloud data fusion device based on joint coding, characterized in that, include: The acquisition unit is used to acquire point cloud data and RGB images of the same target scene collected by the roadside lighthouse; A semantic segmentation unit is used to perform semantic segmentation on the RGB image to obtain a foreground image and a background image, and to determine multiple foreground voxels based on the pixels corresponding to each voxel and the foreground image; The first processing unit is used to perform voxelization processing on point cloud data to obtain multiple voxels that contain at least one point. Transform the center point of the voxel to the camera coordinate system to obtain the pixel corresponding to each voxel; Multiple foreground voxels are determined based on the pixels corresponding to each voxel and the foreground image; The second processing unit is used to obtain the neighboring foreground pixels within the circular area of ​​the foreground pixel corresponding to each foreground voxel, fuse the features of each neighboring foreground pixel to obtain the fused feature, and superimpose the fused feature onto the first feature of the foreground pixel to obtain the second feature of the foreground pixel. The third processing unit is used to superimpose the voxel features of the foreground voxel and the second features of the corresponding foreground pixel to obtain the third feature of the foreground voxel. The third feature of all foreground voxels is processed using a pre-trained self-attention model to obtain the fourth feature of all foreground voxels.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-7.

Images