Point cloud-image bidirectional fusion method, device, vehicle and storage medium

Abstract

The present invention relates to the field of autonomous driving technology, and more specifically to a point cloud-image bidirectional fusion method, apparatus, vehicle, and storage medium. The method comprises obtaining a point cloud feature map and an image feature map for the vehicle's current location, projecting each point in the point cloud feature map onto the coordinate system of the image feature map to obtain homogeneous coordinates for each point in the point cloud feature map, and obtaining a fused image feature map and a fused point cloud feature map based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map. Thus, bidirectional fusion of point cloud data and image data helps improve the efficiency of using environmental data collected by autonomous vehicles, addresses the current problem of being unable to simultaneously process multiple tasks that primarily rely on images and point clouds, and improves the accuracy and reliability of the vehicle's perception of complex environments.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a point cloud-image bidirectional fusion method, device, vehicle, and storage medium. Background Art

[0002] Autonomous driving, as a technology that improves travel efficiency, will significantly improve people's mobility in the near future. Autonomous perception is a key component of autonomous driving technology, providing structured and semantically informed information to decision-making and planning modules. Complex environment perception technology, serving as the "eyes" of intelligent vehicles, lies at the forefront of the autonomous driving system and is the primary safeguard for the safety of both the vehicle and its passengers. To accurately perceive complex and ever-changing traffic environments, intelligent vehicles typically utilize a combination of cameras and lidar. Camera images provide dense texture information, but depth information is lost during the imaging process. LiDAR point clouds provide accurate three-dimensional information about the surrounding environment, but their operating principle dictates that this 3D information is sparse. Images and point clouds contain complementary information, and fusion of these two can provide more accurate information for autonomous driving. However, due to their distinct intrinsic characteristics and data structures, current image-point cloud fusion methods are typically designed for specific tasks.

[0003] In related technologies, patent [CN113887349A] proposes a road area image recognition method based on an image and point cloud fusion network. This method directly inputs the original point cloud into the road area network for fusion with the image, and uses decoding, convolution and other methods to determine whether it is a road area classification category.

[0004] However, this method does not work well in 3D target detection scenarios commonly used in autonomous driving. For some complex scenarios of autonomous driving vehicles, it is often necessary to simultaneously process scene tasks such as road segmentation and scene tasks such as 3D target detection. The one-way fusion method cannot simultaneously meet the multiple application requirements of this scenario, which needs to be solved urgently. Summary of the Invention

[0005] One of the objects of the present invention is to provide a point cloud-image bidirectional fusion method to solve the problem in the prior art that it is impossible to simultaneously process multiple tasks that mainly rely on images and mainly rely on point clouds; the second object is to provide a cloud-image bidirectional fusion device; the third object is to provide a vehicle; and the fourth object is to provide a computer-readable storage medium.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A cloud-image bidirectional fusion method comprises the following steps:

[0008] Obtain the point cloud feature map and image feature map of the vehicle's current location;

[0009] Projecting each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map;

[0010] A fused image feature map and a fused point cloud feature map are obtained according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map.

[0011] According to the above technical means, by bidirectionally fusing point cloud data and image data, vehicles can select appropriate feature maps for environmental perception according to different autonomous driving scenarios, which helps to improve the accuracy and reliability of vehicle perception of complex environments.

[0012] Furthermore, obtaining a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map includes:

[0013] Determining, based on the homogeneous coordinates, an alignment relationship between each point in the point cloud feature map and each pixel in the image feature map;

[0014] Based on the alignment relationship, calculating the feature average of the alignment points of each pixel in the image feature map in the point cloud feature map to obtain a first transferred feature map;

[0015] The first transfer feature map and the image feature map are concatenated to obtain the fused image feature map.

[0016] According to the above technical means, a one-to-one alignment relationship between all points in the point cloud feature map and all pixels in the image feature map can be obtained through homogeneous coordinates. The first transfer feature map and the image feature map obtained according to the alignment relationship are spliced to obtain the fused image feature map.

[0017] Furthermore, the obtaining of a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map further includes:

[0018] Based on the homogeneous coordinates, construct a grid according to the maximum size of the point cloud multi-scale feature map, and map the point cloud feature map features to the grid;

[0019] Extracting image features within the grid to obtain a second transfer feature map;

[0020] The second transferred feature map and the point cloud feature map are concatenated to obtain the fused point cloud feature map.

[0021] According to the above technical means, a grid can be constructed based on the maximum size of the point cloud multi-scale feature map through homogeneous coordinates, and the feature map of the point cloud feature map in the grid is extracted and mapped to the image features of the grid to obtain a second transferred feature map. The second transferred feature map and the point cloud feature map are spliced to obtain a fused point cloud feature map.

[0022] Furthermore, after obtaining the fused image feature map and the fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, the method further includes:

[0023] Obtain current driving scenario requirements;

[0024] If the current driving scene requirement is image information, the fused image feature map is used; if the current driving scene requirement is point cloud information, the fused point cloud feature map is used.

[0025] Based on the above technical means, in complex scenarios of vehicle autonomous driving, corresponding fusion feature maps can be selected according to the requirements of different driving scenarios, which helps to improve the accuracy and reliability of the vehicle's perception of complex environments.

[0026] Furthermore, projecting each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map includes:

[0027] Based on a preset projection formula, each point in the point cloud feature map is projected to the coordinate system of the image feature map to obtain the homogeneous coordinates of each point in the point cloud feature map, wherein the preset projection formula is:

[0028]

[0029] Among them, k i is the scale factor, [u i v i 1] T is the homogeneous coordinate of each point, u i is the horizontal coordinate of the homogeneous coordinate of each point, v i is the ordinate of the homogeneous coordinate of each point, h is the scaling ratio of the image feature map compared to the original image, Q is the camera projection matrix, is the external parameter matrix between the laser radar and the camera, [x i y i z i 1] T is the three-dimensional homogeneous coordinate of the i-th point in the point cloud feature map, x i is the horizontal coordinate of the three-dimensional homogeneous coordinate, yi is the vertical coordinate of the three-dimensional homogeneous coordinate, z i is the vertical coordinate of the three-dimensional homogeneous coordinate.

[0030] According to the above technical means, all points in the point cloud feature map can be projected to the coordinate system of the image feature map according to a preset projection formula to obtain homogeneous coordinates.

[0031] A point cloud-image bidirectional fusion device, comprising:

[0032] An acquisition module is used to obtain a point cloud feature map and an image feature map of the vehicle's current location;

[0033] A projection module, configured to project each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map;

[0034] A fusion module is used to obtain a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map and each pixel in the image feature map.

[0035] Furthermore, the fusion module is specifically used to:

[0036] Determining, based on the homogeneous coordinates, an alignment relationship between each point in the point cloud feature map and each pixel in the image feature map;

[0037] Based on the alignment relationship, calculating the feature average of the alignment points of each pixel in the image feature map in the point cloud feature map to obtain a first transferred feature map;

[0038] The first transfer feature map and the image feature map are concatenated to obtain the fused image feature map.

[0039] Furthermore, the fusion module is further configured to:

[0040] Based on the homogeneous coordinates, construct a grid according to the maximum size of the point cloud multi-scale feature map, and map the point cloud feature map features to the grid;

[0041] Extracting image features within the grid to obtain a second transfer feature map;

[0042] The second transferred feature map and the point cloud feature map are concatenated to obtain the fused point cloud feature map.

[0043] Furthermore, after obtaining the fused image feature map and the fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, the fusion module is further configured to:

[0044] Obtain current driving scenario requirements;

[0045] If the current driving scene requirement is image information, the fused image feature map is used; if the current driving scene requirement is point cloud information, the fused point cloud feature map is used.

[0046] Furthermore, the projection module is specifically used to:

[0047] Based on a preset projection formula, each point in the point cloud feature map is projected to the coordinate system of the image feature map to obtain the homogeneous coordinates of each point in the point cloud feature map, wherein the preset projection formula is:

[0048]

[0049] Among them, k i is the scale factor, [u i v i 1] T is the homogeneous coordinate of each point, u i is the horizontal coordinate of the homogeneous coordinate of each point, v i is the ordinate of the homogeneous coordinate of each point, h is the scaling ratio of the image feature map compared to the original image, Q is the camera projection matrix, is the external parameter matrix between the laser radar and the camera, [x i y i z i 1] T is the three-dimensional homogeneous coordinate of the i-th point in the point cloud feature map, x i is the horizontal coordinate of the three-dimensional homogeneous coordinate, y i is the vertical coordinate of the three-dimensional homogeneous coordinate, z i is the vertical coordinate of the three-dimensional homogeneous coordinate.

[0050] A vehicle includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the point cloud-image bidirectional fusion method as described in the above embodiment.

[0051] A computer-readable storage medium stores a computer program, which is executed by a processor to implement the point cloud-image bidirectional fusion method as described in the above embodiment.

[0052] Beneficial effects of the present invention:

[0053] (1) The present invention can bidirectionally fuse point cloud data and image data, which helps to improve the efficiency of using environmental data collected by autonomous vehicles;

[0054] (2) The present invention allows vehicles to select appropriate feature maps for environmental perception according to different scenarios during autonomous driving, which helps to improve the accuracy and reliability of the vehicle's perception of complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flowchart of a point cloud-image bidirectional fusion method provided according to an embodiment of the present invention;

[0056] Figure 2 is a flowchart of another point cloud-image bidirectional fusion method according to an embodiment of the present invention;

[0057] Figure 3 2 is a block diagram of a point cloud-image bidirectional fusion device according to an embodiment of the present invention;

[0058] Figure 4 FIG. 4 is a schematic structural diagram of a vehicle according to an embodiment of the present invention.

[0059] Among them, 10 is a point cloud-image bidirectional fusion device, 100 is an acquisition module, 200 is a projection module, 300 is a fusion module, 401 is a memory, 402 is a processor, and 403 is a communication interface. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0061] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0062] Figure 1 This is a flowchart of a point cloud-image bidirectional fusion method provided by an embodiment of the present invention.

[0063] Before introducing the point cloud-image bidirectional fusion method proposed in an embodiment of the present invention, a point cloud-image fusion method in related art is briefly introduced.

[0064] In the related art, a target detection and positioning method and system based on image and point cloud fusion is proposed, which includes: determining the edge features of the target to be positioned based on the image, determining the target point cloud of the target to be positioned based on the depth image of the target to be positioned, and performing plane extraction on the point cloud to obtain the plane information of the point cloud, and determining the position of the target to be positioned in the point cloud based on the edge features and plane information of the target to be positioned.

[0065] However, although this method can improve the accuracy of target detection and positioning to a certain extent, it is only applicable to 3D target detection scenarios and is powerless for road segmentation scenarios in autonomous driving.

[0066] It is precisely based on the above problems that an embodiment of the present invention proposes a point cloud-image bidirectional fusion method. By obtaining a point cloud feature map and an image feature map of the vehicle's current location and projecting each point in the point cloud feature map onto the coordinate system of the image feature map, the homogeneous coordinates of each point in the point cloud feature map can be obtained. Based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, a fused image feature map and a fused point cloud feature map are obtained. Therefore, by bidirectionally fusing point cloud data and image data, it helps to improve the efficiency of using environmental data collected by autonomous vehicles, solves the current problem of being unable to simultaneously process multiple tasks that mainly rely on images and mainly rely on point clouds, and improves the accuracy and reliability of the vehicle's perception of complex environments.

[0067] Specifically, if Figure 1 As shown, the point cloud-image bidirectional fusion method includes the following steps:

[0068] In step S101, a point cloud feature map and an image feature map of the current location of the vehicle are obtained.

[0069] Among them, the embodiment of the present invention can use a vehicle-mounted laser radar to obtain a point cloud feature map M of the vehicle's current location, and obtain an image feature map N of the vehicle's current location through a vehicle-mounted camera.

[0070] In step S102, each point in the point cloud feature map is projected to the coordinate system of the image feature map to obtain the homogeneous coordinates of each point in the point cloud feature map.

[0071] Furthermore, in some embodiments, projecting each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map includes: projecting each point in the point cloud feature map to the coordinate system of the image feature map based on a preset projection formula to obtain homogeneous coordinates of each point in the point cloud feature map, wherein the preset projection formula is:

[0072]

[0073] Among them, ki is the scale factor, [u i v i 1] T is the homogeneous coordinate of each point, u i is the abscissa of the homogeneous coordinates of each point, v i is the ordinate of the homogeneous coordinate of each point, h is the scaling ratio of the image feature map compared to the original image, Q is the camera projection matrix, is the external parameter matrix between the laser radar and the camera, [x i y i z i 1] T is the three-dimensional homogeneous coordinate of the i-th point in the point cloud feature map, x i is the horizontal coordinate of the three-dimensional homogeneous coordinate, y i is the vertical coordinate of the three-dimensional homogeneous coordinate, z i is the vertical coordinate of the three-dimensional homogeneous coordinate.

[0074] Specifically, obtain the external parameter matrix between the vehicle-mounted lidar and the camera in, Get the vehicle camera projection matrix Q∈R 3×4 ; According to the preset projection formula, each point M in the point cloud feature map M i Projected into the image feature map N coordinate system, the homogeneous coordinates of each point after projection [u i v i 1] T .

[0075] In step S103, a fused image feature map and a fused point cloud feature map are obtained according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map.

[0076] Furthermore, in some embodiments, a fused image feature map and a fused point cloud feature map are obtained based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, including: determining the alignment relationship between each point in the point cloud feature map and each pixel in the image feature map based on the homogeneous coordinates; calculating the feature average value of the alignment point in the point cloud feature map for each pixel in the image feature map based on the alignment relationship to obtain a first transfer feature map; and splicing the first transfer feature map and the image feature map to obtain a fused image feature map.

[0077] Specifically, each point M in the point cloud feature map M is first determined by the following formula i Align pixel n in image feature map N i , thereby determining each point M in the point cloud feature map M i With each pixel n in the image feature map i alignment relationship.

[0078]

[0079] in, Indicates rounding down.

[0080] Secondly, each pixel n is determined by the following formula i The average value n of the features of the aligned points in the point cloud feature map M i ', aggregate all transfer features n i ', and obtain the first transfer feature map N'.

[0081]

[0082] Among them, U(n i ) is each point M in the point cloud feature map M i With pixel n i The set of aligned points, R i U(n i ) number of midpoints, j is the number of midpoints in U(n i ), an element in j is the jth point in the point cloud feature map M.

[0083] Finally, the first transfer feature map N' is spliced with the image feature map N to obtain the fused image feature map N θ .

[0084] Furthermore, in some embodiments, a fused image feature map and a fused point cloud feature map are obtained based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, and the method further includes: constructing a grid according to the maximum size of the point cloud multi-scale feature map based on the homogeneous coordinates, and mapping the point cloud feature map features to the grid; extracting image features within the grid to obtain a second transferred feature map; and splicing the second transferred feature map and the point cloud feature map to obtain a fused point cloud feature map.

[0085] Specifically, in the projection coordinates (i.e., homogeneous coordinates) of the point cloud feature map M to the image feature map N, a grid is constructed according to the maximum size of the point cloud multi-scale feature map, and bilinear interpolation is performed according to the following formula to map the image features to the grid, obtaining M i The transfer feature M i '.

[0086] M′ i =∑(1-|u i -j|)·(1-|v i -k|)·N jk ;

[0087]

[0088] in, Indicates rounding up, N jk is the jk-th pixel in the image feature map N.

[0089] For each point M in the point cloud feature map M i By performing the above calculations, we can obtain the high-dimensional transfer feature M for each point. i ', forming a transfer feature map; further extract the image features in the grid using a neural network algorithm to obtain a second transfer feature map M'; splice the second transfer feature map M' and the point cloud feature map M to obtain the fused point cloud feature map M θ .

[0090] In addition, in some embodiments, after obtaining the fused image feature map and the fused point cloud feature map based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, it also includes: obtaining the current driving scene requirements; if the current driving scene requirement is image information, the fused image feature map is used; if the current driving scene requirement is point cloud information, the fused point cloud feature map is used.

[0091] It is understandable that N θ With M θ It is the feature map after the bidirectional fusion of image and point cloud. In the complex scene of vehicle autonomous driving, different fusion feature maps can be selected for environmental perception based on the current different driving scene requirements. If the current driving scene requires image information, that is, to solve problems such as road area segmentation that are mainly based on image information, the fused image feature map N can be used. θ For environmental perception; if the current driving scene requires point cloud information, that is, to solve problems such as 3D target detection based on point cloud information, the fused point cloud feature map M can be used θ Conduct environmental perception.

[0092] To facilitate those skilled in the art to further understand the point cloud-image bidirectional fusion method proposed in the embodiment of the present invention, the following is a Figure 2 Provide further explanation.

[0093] Figure 2 FIG. 1 is a flow chart of another point cloud-image bidirectional fusion method according to an embodiment of the present invention. Figure 2 As shown, the method includes the following steps:

[0094] Step S201a, obtaining a point cloud feature map M.

[0095] Step S201b: Obtain image feature map N.

[0096] Step S202: Project each point in the point cloud feature map M to the coordinate system of the image feature map N to obtain the homogeneous coordinates of each point in the projected point cloud feature map M. Execute steps S203a and S204a at the same time.

[0097] Step S203a: construct a grid and perform bilinear interpolation to map the image features into the grid.

[0098] Step S203b: extract features using a neural network algorithm to determine a transfer feature map of M.

[0099] In step S203c, the transferred feature map of M is concatenated with M to obtain a fused point cloud feature map.

[0100] Step S204a: determining the alignment relationship between each point in the point cloud feature map M and each pixel in the image feature map N.

[0101] Step S204b, calculating the feature average of each pixel in the image feature map N at the aligned points in the point cloud feature map M, and determining the transfer feature map of N.

[0102] In step S204c, the transferred feature map of N is concatenated with N to obtain a fused image feature map.

[0103] Step S205: Select and use the corresponding fusion feature map according to different question topics.

[0104] According to the point cloud-image bidirectional fusion method proposed in an embodiment of the present invention, by obtaining a point cloud feature map and an image feature map of the vehicle's current location, and projecting each point in the point cloud feature map onto the coordinate system of the image feature map, the homogeneous coordinates of each point in the point cloud feature map can be obtained. Based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, a fused image feature map and a fused point cloud feature map are obtained. Thus, by bidirectionally fusing point cloud data and image data, the efficiency of using environmental data collected by autonomous vehicles can be improved, solving the current problem of being unable to simultaneously process multiple tasks that primarily rely on images and primarily on point clouds, and improving the accuracy and reliability of the vehicle's perception of complex environments.

[0105] Next, a point cloud-image bidirectional fusion device according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0106] Figure 3 4 is a block diagram of a point cloud-image bidirectional fusion device according to an embodiment of the present invention.

[0107] like Figure 3 As shown, the point cloud-image bidirectional fusion device 10 includes: an acquisition module 100 , a projection module 200 and a fusion module 300 .

[0108] The acquisition module 100 is used to obtain a point cloud feature map and an image feature map of the current location of the vehicle;

[0109] A projection module 200 is used to project each point in the point cloud feature map to the coordinate system of the image feature map to obtain the homogeneous coordinates of each point in the point cloud feature map;

[0110] The fusion module 300 is used to obtain a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map.

[0111] Furthermore, in some embodiments, the fusion module 300 is specifically configured to:

[0112] Based on homogeneous coordinates, determine the alignment relationship between each point in the point cloud feature map and each pixel in the image feature map;

[0113] Based on the alignment relationship, the feature average of the alignment points of each pixel in the image feature map in the point cloud feature map is calculated to obtain the first transferred feature map;

[0114] The first transferred feature map and the image feature map are concatenated to obtain a fused image feature map.

[0115] Furthermore, in some embodiments, the fusion module 300 is further configured to:

[0116] Based on homogeneous coordinates, a grid is constructed according to the maximum size of the point cloud multi-scale feature map, and the features of the point cloud feature map are mapped to the grid;

[0117] Extracting image features within the grid to obtain a second transferred feature map;

[0118] The second transferred feature map and the point cloud feature map are concatenated to obtain a fused point cloud feature map.

[0119] Furthermore, in some embodiments, after obtaining the fused image feature map and the fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, the fusion module 300 is further configured to:

[0120] Obtain current driving scenario requirements;

[0121] If the current driving scene requires image information, the fused image feature map is used. If the current driving scene requires point cloud information, the fused point cloud feature map is used.

[0122] Furthermore, in some embodiments, the projection module 200 is specifically configured to:

[0123] Based on the preset projection formula, each point in the point cloud feature map is projected to the coordinate system of the image feature map to obtain the homogeneous coordinates of each point in the point cloud feature map. The preset projection formula is:

[0124]

[0125] Among them, k i is the scale factor, [u i v i 1] T is the homogeneous coordinate of each point, u i is the abscissa of the homogeneous coordinates of each point, v i is the ordinate of the homogeneous coordinate of each point, h is the scaling ratio of the image feature map compared to the original image, Q is the camera projection matrix, is the external parameter matrix between the laser radar and the camera, [x i y i z i 1] T is the three-dimensional homogeneous coordinate of the i-th point in the point cloud feature map, x i is the horizontal coordinate of the three-dimensional homogeneous coordinate, y i is the vertical coordinate of the three-dimensional homogeneous coordinate, z i is the vertical coordinate of the three-dimensional homogeneous coordinate.

[0126] It should be noted that the aforementioned explanation of the point cloud-image bidirectional fusion method embodiment is also applicable to the point cloud-image bidirectional fusion device of this embodiment, and will not be repeated here.

[0127] According to the point cloud-image bidirectional fusion device proposed in an embodiment of the present invention, by obtaining a point cloud feature map and an image feature map of the vehicle's current location, and projecting each point in the point cloud feature map onto the coordinate system of the image feature map, the homogeneous coordinates of each point in the point cloud feature map can be obtained. Based on the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, a fused image feature map and a fused point cloud feature map are obtained. Thus, by bidirectionally fusing point cloud data and image data, the efficiency of using environmental data collected by autonomous vehicles can be improved, solving the current problem of being unable to simultaneously process multiple tasks that primarily rely on images and primarily on point clouds, and improving the accuracy and reliability of the vehicle's perception of complex environments.

[0128] Figure 4 A schematic diagram of the structure of a vehicle provided in an embodiment of the present invention. The vehicle may include:

[0129] Memory 401 , processor 402 , and computer programs stored in the memory 401 and executable on the processor 402 .

[0130] When the processor 402 executes the program, the point cloud-image bidirectional fusion method provided in the above embodiment is implemented.

[0131] Furthermore, the vehicle further comprises:

[0132] The communication interface 403 is used for communication between the memory 401 and the processor 402 .

[0133] The memory 401 is used to store computer programs that can be run on the processor 402 .

[0134] The memory 401 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0135] If the memory 401, the processor 402, and the communication interface 403 are implemented independently, the communication interface 403, the memory 401, and the processor 402 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0136] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0137] The processor 402 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0138] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above point cloud-image bidirectional fusion method when executed by a processor.

[0139] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or modification made by those skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. A point cloud-image bidirectional fusion method, characterized in that: The following steps are involved: Obtain the point cloud feature map and image feature map of the vehicle's current location; Projecting each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map; Obtaining a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map; After obtaining the fused image feature map and the fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map, the method further includes: Obtain current driving scenario requirements; If the current driving scene requirement is image information, the fused image feature map is used; if the current driving scene requirement is point cloud information, the fused point cloud feature map is used.

2. The method according to claim 1, characterized in that The obtaining a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map includes: Determining, based on the homogeneous coordinates, an alignment relationship between each point in the point cloud feature map and each pixel in the image feature map; Based on the alignment relationship, calculating the feature average of the alignment points of each pixel in the image feature map in the point cloud feature map to obtain a first transferred feature map; The first transfer feature map and the image feature map are concatenated to obtain the fused image feature map.

3. The method according to claim 1, characterized in that The step of obtaining a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map, and each pixel in the image feature map further includes: Based on the homogeneous coordinates, construct a grid according to the maximum size of the point cloud multi-scale feature map, and map the image features to the grid; Extracting image features within the grid to obtain a second transfer feature map; The second transferred feature map and the point cloud feature map are concatenated to obtain the fused point cloud feature map.

4. The method according to claim 1, wherein Projecting each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map includes: Based on a preset projection formula, each point in the point cloud feature map is projected to the coordinate system of the image feature map to obtain the homogeneous coordinates of each point in the point cloud feature map, wherein the preset projection formula is: Among them, k i is the scale factor, [u i v i 1] T is the homogeneous coordinate of each point, u i is the horizontal coordinate of the homogeneous coordinate of each point, v i is the ordinate of the homogeneous coordinate of each point, h is the scaling ratio of the image feature map compared to the original image, Q is the camera projection matrix, is the external parameter matrix between the laser radar and the camera, [x i y i z i 1] T is the three-dimensional homogeneous coordinate of the i-th point in the point cloud feature map, x i is the horizontal coordinate of the three-dimensional homogeneous coordinate, y i is the vertical coordinate of the three-dimensional homogeneous coordinate, z i is the vertical coordinate of the three-dimensional homogeneous coordinate.

5. A point cloud-image bidirectional fusion device, characterized in that: A method for implementing the point cloud-image bidirectional fusion method according to any one of claims 1 to 4, comprising: An acquisition module is used to obtain a point cloud feature map and an image feature map of the vehicle's current location; A projection module, configured to project each point in the point cloud feature map to the coordinate system of the image feature map to obtain homogeneous coordinates of each point in the point cloud feature map; A fusion module is used to obtain a fused image feature map and a fused point cloud feature map according to the homogeneous coordinates, each point in the point cloud feature map and each pixel in the image feature map.

6. The device according to claim 5, characterized in that The fusion module is specifically used to: Determining, based on the homogeneous coordinates, an alignment relationship between each point in the point cloud feature map and each pixel in the image feature map; Based on the alignment relationship, calculating the feature average of the alignment points of each pixel in the image feature map in the point cloud feature map to obtain a first transferred feature map; The first transfer feature map and the image feature map are concatenated to obtain the fused image feature map.

7. The device according to claim 5, characterized in that The fusion module is further configured to: Based on the homogeneous coordinates, construct a grid according to the maximum size of the point cloud multi-scale feature map, and map the image features to the grid; Extracting image features within the grid to obtain a second transfer feature map; The second transferred feature map and the point cloud feature map are concatenated to obtain the fused point cloud feature map.

8. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the point cloud-image bidirectional fusion method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the point cloud-image bidirectional fusion method according to any one of claims 1 to 4.

Images