A feature map fusion method and device

Abstract

A feature map fusion method and device are disclosed, the method comprising: obtaining a laser radar image corresponding to a three-dimensional point cloud collected by a laser radar; obtaining a camera perspective map corresponding to the three-dimensional point cloud under the camera perspective view according to the three-dimensional point cloud and camera parameters; obtaining a conversion feature map of the laser radar image under the camera perspective view according to the correspondence between the laser radar image and the camera perspective map; fusing the camera perspective map and the conversion feature map to obtain a fused feature map. The present invention utilizes the camera perspective map under the camera perspective view and the rich semantic information of the three-dimensional point cloud to fuse the laser radar image with the camera perspective map. The obtained feature map has the characteristics of both the camera feature map and the radar feature map, thereby improving the quality of the obtained feature map and helping to improve the detection effect of using the feature map to assist road detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a feature map fusion method and device. Background Art

[0002] Road detection is a very important part of the field of autonomous driving. In order to obtain high-precision road detection results, people have proposed a variety of methods. A major difference between each method is that the sensor used to collect road images is different, for example, some use lidar, and some use monocular cameras.

[0003] Monocular cameras can collect data at high speed, and the collected images contain very rich information. However, they are greatly affected by the environment. For example, under extreme lighting conditions or extreme weather conditions, it is impossible to obtain satisfactory images through monocular cameras, resulting in poor road detection results. In addition, due to the large amount of information flow of images collected by monocular cameras, it is necessary to build a complex model to process them. The processing process takes a lot of time, making it difficult to perform real-time road detection.

[0004] LiDAR can provide very accurate geometric information, but it can only obtain sparse point cloud information, and the point cloud distribution is very irregular, resulting in poor road detection effect. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a feature map fusion method, device, computer-readable storage medium and electronic device, which acquire data through a laser radar, and then process the laser radar data to obtain a laser radar image and a camera perspective image under the camera perspective view, and fuse the two images to obtain a feature map, which can greatly improve the quality of the obtained feature map, thereby helping to improve the detection effect of using the feature map to assist road detection.

[0006] According to a first aspect of the present application, a feature map fusion method is provided, comprising:

[0007] Obtain a laser radar image corresponding to the three-dimensional point cloud collected by the laser radar;

[0008] According to the three-dimensional point cloud and the camera parameters, a camera perspective view corresponding to the three-dimensional point cloud under the camera perspective angle is obtained;

[0009] According to the correspondence between the laser radar image and the camera perspective view, a conversion feature map of the laser radar image under the camera perspective view is obtained;

[0010] The camera perspective image and the conversion feature map are fused to obtain a fused feature map.

[0011] According to a second aspect of the present application, a feature map fusion device is provided, comprising:

[0012] A radar image acquisition module, used to acquire a laser radar image corresponding to the three-dimensional point cloud according to the three-dimensional point cloud collected by the laser radar;

[0013] A camera perspective image acquisition module is used to acquire a camera perspective image corresponding to the three-dimensional point cloud under the camera perspective angle according to the three-dimensional point cloud collected by the laser radar;

[0014] A conversion feature map acquisition module, used to acquire a conversion feature map of the laser radar image under the camera perspective view according to the laser radar image;

[0015] A fusion module is used to fuse the camera perspective image and the conversion feature map to obtain a fused feature map.

[0016] According to a third aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above-mentioned feature map fusion method.

[0017] According to a fourth aspect of the present application, an electronic device is provided, the electronic device comprising:

[0018] processor;

[0019] a memory for storing instructions executable by the processor;

[0020] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned feature map fusion method.

[0021] Compared with the prior art, the feature map fusion method, device, computer-readable storage medium and electronic device provided by the present application include at least the following beneficial effects: the present application constructs a virtual camera to project the three-dimensional point cloud obtained by the lidar under the camera perspective view to obtain the corresponding camera perspective view, and at the same time makes full use of the rich semantic information of the three-dimensional point cloud to fuse the lidar image with the camera perspective view. The obtained feature map has the characteristics of both the camera feature map and the radar feature map, which greatly improves the quality of the obtained feature map, and thus helps to improve the detection effect of using the feature map to assist road detection, thereby making the road detection more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 This is a schematic diagram of a feature map fusion method provided by an exemplary embodiment of the present application. Figure 1 .

[0024] Figure 2 Yes Figure 1 The flowchart of step 10 in the embodiment shown is shown.

[0025] Figure 3 It is a schematic diagram of obtaining a lidar image in a feature map fusion method provided by an exemplary embodiment of the present application, wherein the first row is a depth image in the X direction, the second row is a width image in the Y direction, the third row is a height image in the Z direction, and the fourth row is a distribution intensity image of data points.

[0026] Figure 4 Yes Figure 1 The flowchart of step 20 in the embodiment shown is shown.

[0027] Figure 5 Yes Figure 4 The flowchart of step 202 in the embodiment shown is shown.

[0028] Figure 6 Yes Figure 4 The flowchart of step 203 in the embodiment shown is shown.

[0029] Figure 7 Yes Figure 1 The flowchart of step 30 in the embodiment shown is shown.

[0030] Figure 8 (1) is a schematic diagram of the mapping layer. Figure 8 (2) is a schematic diagram of the fusion layer.

[0031] Fig. 9 This is a schematic diagram of a feature map fusion method provided by an exemplary embodiment of the present application. Figure 2 .

[0032] Fig.10 This is a schematic diagram of a feature graph fusion device provided by an exemplary embodiment of the present application. Figure 1 .

[0033] Fig.11It is a schematic diagram of a radar image acquisition module in a feature map fusion device provided by an exemplary embodiment of the present application.

[0034] Fig.12 It is a schematic diagram of a camera perspective image acquisition module in a feature image fusion device provided by an exemplary embodiment of the present application.

[0035] Fig.13 This is a schematic diagram of a feature graph fusion device provided by an exemplary embodiment of the present application. Figure 2 .

[0036] Fig.14 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0037] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0038] Application Overview

[0039] When a vehicle is driving, it is necessary to perform real-time road detection so that the road conditions can be known in time. In order to obtain high-precision road detection results, it is necessary to collect road images in real time. Currently, the commonly used image collection methods are laser radar image collection and ordinary camera image collection.

[0040] LiDAR images can provide very accurate geometric information, but LiDAR can only obtain sparse point cloud information, and the point cloud distribution is very irregular, resulting in poor results when using LiDAR data for road detection. Ordinary cameras (such as monocular cameras) are becoming more and more popular because they are cheap, can perform high-speed continuous acquisition, and can collect very rich details of the images. However, monocular cameras also have their inherent disadvantages, such as being greatly affected by the environment. In some extreme lighting conditions or weather conditions (such as rain, cloudy days, etc.), satisfactory images cannot be obtained, so road detection based on these images is not effective.

[0041] The present invention proposes a new feature map fusion method, which makes full use of the fact that the data obtained by the laser radar is not affected by extreme weather conditions, etc., obtains data through the laser radar, and then processes the laser radar data to obtain the laser radar image and the camera perspective image under the camera perspective (virtual camera), and then fuses the two images to obtain the feature map, which can greatly improve the quality of the obtained feature map, and then help improve the detection effect of using the feature map to assist road detection, so that the accuracy of road detection is higher.

[0042] After introducing the basic concept of the present disclosure, various non-limiting embodiments of the technical solutions provided in the present application will be specifically introduced in conjunction with the accompanying drawings.

[0043] Exemplary Methods

[0044] Figure 1 It is a flowchart of a feature map fusion method provided by an exemplary embodiment of the present application.

[0045] This embodiment can be applied to electronic devices, and specifically to autonomous driving terminals. Figure 1 As shown, a feature map fusion method provided by an exemplary embodiment of the present application includes at least the following steps:

[0046] Step 10: Obtain the lidar image corresponding to the three-dimensional point cloud collected by the lidar.

[0047] Since laser radar is not affected by extreme weather conditions and can obtain very accurate images under any circumstances, this embodiment uses laser radar to collect three-dimensional point clouds of road images. The three-dimensional point cloud contains multiple data points arranged in disorder, so the distribution of data points is very irregular; this embodiment processes the coordinates of each data point in the three-dimensional point cloud to make the expression of the data points more systematic and organized, thereby forming a laser radar image corresponding to the three-dimensional point cloud.

[0048] Step 20: According to the three-dimensional point cloud and the camera parameters, obtain the camera perspective view corresponding to the three-dimensional point cloud under the camera perspective angle.

[0049] Considering that the 3D point cloud data collected by the laser radar is relatively sparse, if only the laser radar image corresponding to the 3D point cloud is used as input, the obtained feature map effect is not good. Therefore, this embodiment further processes the 3D point cloud data collected by the laser radar. This embodiment constructs a virtual camera, and the internal parameters of the virtual camera and the external parameters between the laser radar are directly specified, so that the data points in the 3D point cloud can be projected onto the image plane of the virtual camera, and the camera perspective view (also called camera feature map) of the 3D point cloud under the camera perspective view is obtained.

[0050] Step 30: According to the correspondence between the lidar image and the camera perspective view, obtain a conversion feature map of the lidar image under the camera perspective view.

[0051] Since there is a projection relationship between the virtual camera and the laser radar, the data points in the laser radar image can be further processed, and the feature map to be converted in the laser radar image is converted to the camera perspective to obtain the corresponding conversion feature map (also called radar feature map). The feature map to be converted here corresponds to the part of interest in the laser radar image, for example, the road surface part in the laser radar image; of course, it can also correspond to other parts, which is not limited here.

[0052] Step 40: Fuse the camera perspective image and the conversion feature map to obtain a fused feature map.

[0053] In the camera perspective view and conversion feature map corresponding to the three-dimensional point cloud obtained by the lidar, the camera perspective view is equivalent to the image obtained by a virtual camera, and the feature conversion map is equivalent to the image obtained by the lidar, which contains very rich semantic information. Therefore, after the camera perspective view and the conversion feature map are fused, the obtained feature map has the characteristics of both the camera perspective view and the conversion feature map, which greatly improves the quality of the obtained feature map.

[0054] The feature map obtained in this embodiment can be applied to various scenarios as needed. For example, the feature map can be used in road detection. At this time, the rich semantic information possessed by the converted feature map can assist road detection under the camera perspective, effectively improving the accuracy of road detection.

[0055] The beneficial effect of the feature map fusion method provided in this embodiment is at least that: this embodiment constructs a virtual camera to project the three-dimensional point cloud obtained by the lidar under the camera perspective view to obtain the corresponding camera perspective view, and at the same time makes full use of the rich semantic information of the three-dimensional point cloud to fuse the lidar image with the camera perspective view. The obtained feature map has the characteristics of both the camera feature map and the radar feature map, which greatly improves the quality of the obtained feature map, and thus helps to improve the detection effect of using the feature map to assist road detection, thereby making the road detection more accurate.

[0056] Figure 2 It is a flowchart of step 10 in the feature map fusion method provided by an exemplary embodiment of the present application.

[0057] like Figure 2 As shown, obtaining a laser radar image corresponding to a three-dimensional point cloud provided by an exemplary embodiment of the present application includes at least the following steps:

[0058] Step 101: Collect images through laser radar to obtain a three-dimensional point cloud of the image.

[0059] In this embodiment, a laser radar is used to collect road images in real time. The collected images are three-dimensional point cloud images, including multiple scattered data points, each of which has three-dimensional coordinate information, denoted as P v (x v ,y v ,z v ).

[0060] Step 102: Obtain the laser radar image corresponding to the three-dimensional point cloud in the polar coordinate system.

[0061] Through polar coordinate transformation, each data point in the three-dimensional point cloud can be transformed to obtain a lidar image in polar coordinate form, thereby making the scattered data points in the three-dimensional point cloud more orderly and the expression more systematic.

[0062] In this embodiment, the three-dimensional coordinates of the laser radar and the polar coordinates (angle θ, angle and distance p i ) can be converted into:

[0063] θ=atan2(y v ,x v ) Formula (1)

[0064]

[0065]

[0066] Among them, atan2(y v ,x v ) = arctan(y v / x v ), angle θ and are the horizontal angle and the vertical angle respectively; p i is the distance between the data point in the lidar image and the origin of the polar coordinates; δθ and are the horizontal resolution and the vertical resolution, respectively. In this embodiment, the horizontal resolution δθ of each pixel can be set to 0.25°. After the conversion from the above formula (1) to formula (3), a value of H can be obtained. i ×W i ×D i A 3D tensor, where H i and W i are the height and width of the lidar image (can be 64 and 360), respectively. i is the dimension of the data point in the LiDAR image. In this embodiment, D i is 4 (including 3 coordinate axes and 1 intensity value). In one embodiment, after the 3D point cloud is processed, the obtained laser radar image can be as follows Figure 3As shown, the first line is a depth image in the X direction, the second line is a width image in the Y direction, the third line is a height image in the Z direction, and the fourth line is a distribution intensity image of data points. The black area indicates that there is no corresponding data point in the three-dimensional point cloud.

[0067] This embodiment converts the three-dimensional point cloud into polar coordinates, so that the expression of scattered data points in the three-dimensional point cloud can be more systematic, thereby obtaining a lidar image with more orderly data points.

[0068] Figure 4 It is a flowchart of step 20 in the feature map fusion method provided by an exemplary embodiment of the present application.

[0069] like Figure 4 As shown, an exemplary embodiment of the present application provides obtaining a camera perspective view corresponding to a three-dimensional point cloud under a camera perspective view, which includes at least the following steps:

[0070] Step 201: Obtain first coordinate information of an image captured by the laser radar in the laser radar coordinate system.

[0071] The first coordinate information of the data point in the 3D point cloud image collected by the laser radar can be expressed as P v (x v ,y v ,z v ).

[0072] Step 202: Obtain second coordinate information corresponding to the first coordinate information in the camera coordinate system.

[0073] In this embodiment, the camera coordinate system is the coordinate system corresponding to the virtual camera, which can be generated in a specified manner. Therefore, the conversion relationship between the camera coordinate system and the lidar coordinate system (external parameters of the virtual camera) can be generated in a preset manner, and the first coordinate information of the data point can obtain the second coordinate information under the camera coordinate system according to the conversion relationship.

[0074] Step 203: Obtain pixel coordinate information corresponding to the second coordinate information in the image coordinate system to obtain a camera perspective view.

[0075] In this embodiment, the internal parameters of the virtual camera can be generated in a specified manner, so the conversion relationship between the camera coordinate system and the image coordinate system can be obtained in a preset manner, and the second coordinate information of the data point can obtain the pixel coordinate information in the image coordinate system according to the conversion relationship, and all the pixel coordinate information constitutes the camera perspective view. It should be understood that the image coordinate system is a plane system, so when the image corresponding to the data point is projected from the camera coordinate system to the image coordinate system, the depth information of the image will be lost.

[0076] This embodiment constructs a virtual camera and specifies the external parameters and internal parameters of the virtual camera, so that the three-dimensional point cloud data can be projected onto the image plane of the virtual camera to obtain a camera perspective view.

[0077] Figure 5 It is a flowchart of step 202 in the feature map fusion method provided by an exemplary embodiment of the present application.

[0078] like Figure 5 As shown, an exemplary embodiment of the present application provides obtaining second coordinate information corresponding to first coordinate information in a camera coordinate system, including the following steps:

[0079] Step 2021: Obtain the transformation matrix and correction matrix of the laser radar coordinate system and the camera coordinate system. The transformation matrix and correction matrix are respectively and R rect , which can be generated in a specified way.

[0080] Step 2022: Obtain the second coordinate information P corresponding to the first coordinate information in the camera coordinate system according to the transformation matrix and the correction matrix c (x c ,y c ,z c ). In this embodiment, the second coordinate information may be expressed as:

[0081]

[0082] This embodiment constructs a virtual camera and specifies the external parameters of the virtual camera, so as to obtain the coordinate information of the three-dimensional point cloud data in the camera coordinate system, thereby realizing the conversion of the three-dimensional point cloud data from the lidar coordinate system to the camera coordinate system.

[0083] Figure 6 It is a flowchart of step 203 in the feature map fusion method provided by an exemplary embodiment of the present application.

[0084] like Figure 6 As shown, an exemplary embodiment of the present application provides obtaining pixel coordinate information corresponding to the second coordinate information in the image coordinate system, including the following steps:

[0085] Step 2031: Obtain the projection matrix of the camera coordinate system and the image coordinate system. The projection matrix of this embodiment is T proj , which can be generated in a specified way.

[0086] Step 2032: Obtain pixel coordinate information p corresponding to the second coordinate information in the image coordinate system according to the projection matrix c (μ c ,νc ). In this embodiment, the pixel coordinate information may be expressed as:

[0087]

[0088] This embodiment constructs a virtual camera and specifies the internal parameters of the virtual camera, so as to obtain the coordinate information of the three-dimensional point cloud data in the pixel coordinate system, and realizes the conversion of the three-dimensional point cloud data from the camera coordinate system to the pixel coordinate system. After the conversion from the above formula (4) to formula (5), a size H can be obtained. c ×W c ×D c A 3D tensor, where H c and W c are the height and width of the camera perspective view, D c is the dimension of the data point in the LiDAR image. In this embodiment, D i It is 4 (including 3 coordinate axes and 1 intensity value).

[0089] Figure 7 It is a flowchart of step 30 in the feature map fusion method provided by an exemplary embodiment of the present application.

[0090] like Figure 7 As shown, an exemplary embodiment of the present application provides a method for obtaining a conversion feature map of a laser radar image under a camera perspective, including the following steps:

[0091] Step 301: Obtain a mapping diagram based on the correspondence between the lidar image and the camera perspective view.

[0092] The generation of the laser radar image and the camera perspective view both involve the coordinate information of the data points (also called radar points) in the three-dimensional point cloud collected by the laser radar. Here, a mapping layer can be constructed to establish a corresponding relationship between the laser radar image and the camera perspective view using the coordinate information of the data points. In this embodiment, the feature map to be converted in the laser radar image is considered to be F i Indicates that its size is H i ×W i ×D i , the feature map F in the camera perspective map that needs to be transformed i The corresponding mapping graph P is represented by, and its size is H i ×W i , then the relationship between the coordinates of the same radar point in the lidar image and the projected coordinates in the camera perspective image can be as follows:

[0093]

[0094] where p i =(ui , v i )∈I i and p c =(u c , v c )∈I c They represent the projection coordinates of the same radar point in the radar image and the camera perspective view, respectively, and width is the width of the camera perspective view. From the above formula (6), it can be seen that the mapping map P stores the correspondence between the above lidar image and the camera perspective view.

[0095] Step 302: Obtain a feature conversion map based on the mapping map and the LiDAR image.

[0096] In this embodiment, the feature conversion graph F c The expression can be:

[0097] F c =f scatter (F i ,P) Formula (7)

[0098] Among them, f scatter is a mapping function, which may be a Scatter function.

[0099] According to the above formula (7), we can get the size H under the camera perspective c ×W c ×D c The conversion feature map F c , so that the feature map F extracted from the lidar image can be i Image resolution scaled to the camera's perspective. Figure 8 (1), for example, the feature map size extracted from the lidar image is 64×360×4, while the input of the camera image subnetwork needs to be 376×1248. By using the above formula (7), the feature map F i The size of the image is converted to 376 × 1248 × 4. Considering the size of the lidar image, the overall computational complexity of the above transformation process is very small.

[0100] Furthermore, after obtaining the conversion feature map, in order to make full use of the rich semantic information of the lidar image, this embodiment can fuse the conversion feature map with the camera perspective image by constructing a fusion layer. Please refer to Figure 8 (2), for example, fusion can be achieved through the CONCATENATE function, thereby realizing a fusion layer that combines image information from two perspectives.

[0101] Furthermore, after obtaining the feature map, this embodiment can further apply it to road detection. Fig. 9It is another flowchart of a feature map fusion method provided by an exemplary embodiment of the present application.

[0102] like Fig. 9 As shown, the feature map fusion method provided by an exemplary embodiment of the present application also includes the following steps:

[0103] Step 50: Input the fused feature map into the camera perspective subnetwork to assist road detection under the camera perspective.

[0104] Considering the differences between different feature maps, late fusion is a more direct choice. However, when performing road detection, the road detection result using only the camera perspective map is not ideal. Therefore, this embodiment adopts the early fusion method, that is, the camera perspective map is fused with the conversion feature map to form a feature map, and then the feature map is used for road detection. At this time, the conversion feature map with rich semantic information can assist road detection under the camera perspective view, which can effectively improve the road detection effect.

[0105] Exemplary Devices

[0106] Based on the same concept as the method embodiment of the present application, the embodiment of the present application also provides a feature map fusion device.

[0107] Fig.10 A schematic diagram of the structure of a feature graph fusion device provided by an exemplary embodiment of the present application is shown.

[0108] The feature map fusion device provided in this embodiment includes a radar image acquisition module 61, a camera perspective acquisition module 62, a conversion feature map acquisition module 63 and a fusion module 64. Among them, the radar image acquisition module 61 is used to obtain the laser radar image corresponding to the three-dimensional point cloud collected by the laser radar; the camera perspective acquisition module 62 is used to obtain the camera perspective view corresponding to the three-dimensional point cloud under the camera perspective view according to the three-dimensional point cloud and camera parameters; the conversion feature map acquisition module 63 is used to obtain the conversion feature map of the laser radar image under the camera perspective view according to the corresponding relationship between the laser radar image and the camera perspective view; the fusion module 64 is used to fuse the camera perspective view and the conversion feature map to obtain a fused feature map.

[0109] See also Fig.11 Further, the radar image acquisition module 61 includes an image acquisition unit 611 and a radar image acquisition unit 612. The image acquisition unit 611 is used to acquire images and obtain a three-dimensional point cloud of the image; the radar image acquisition unit 612 is used to acquire a laser radar image corresponding to the three-dimensional point cloud in a polar coordinate system.

[0110] See also Fig.12Further, the camera perspective image acquisition module 62 includes a first coordinate acquisition unit 621, a second coordinate acquisition unit 622 and a pixel coordinate acquisition unit 623. The first coordinate acquisition unit 621 is used to acquire the first coordinate information of the image collected by the laser radar in the laser radar coordinate system, the second coordinate acquisition unit 622 is used to acquire the second coordinate information corresponding to the first coordinate information in the camera coordinate system, and the pixel coordinate acquisition unit 623 is used to acquire the pixel coordinate information corresponding to the second coordinate information in the image coordinate system.

[0111] See also Fig.13 ,Furthermore, the feature map fusion device also includes a road detection module 65, which is used to input the fused feature map into the camera perspective sub-network to assist road detection under the camera perspective view.

[0112] Exemplary Electronic Devices

[0113] Fig.14 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0114] like Fig.14 As shown, the electronic device 700 includes one or more processors 701 and a memory 702 .

[0115] The processor 701 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 700 to perform desired functions.

[0116] The memory 702 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 701 may execute the program instructions to implement the feature map fusion method of each embodiment of the present application described above and / or other desired functions.

[0117] In one example, the electronic device 700 may further include: an input device 703 and an output device 704, and these components are interconnected via a bus system and / or a connection mechanism (not shown) in other forms.

[0118] The input device 703 may include, for example, a keyboard, a mouse, etc.

[0119] The output device 704 can output various information to the outside, including the determined distance information, direction information, etc. The output device 704 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0120] Of course, to simplify, Fig.14 Only some of the components related to the present application in the electronic device 700 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 700 may also include any other appropriate components.

[0121] Exemplary computer program products and computer-readable storage media

[0122] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the feature map fusion method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0123] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the feature map fusion method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0125] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0127] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0128] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0129] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0130] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A feature map fusion method, comprising: Obtain a laser radar image corresponding to the three-dimensional point cloud collected by the laser radar; According to the three-dimensional point cloud and the virtual camera parameters, a camera perspective view corresponding to the three-dimensional point cloud under the camera perspective angle is obtained, which includes: projecting the data points of the three-dimensional point cloud onto the image plane of the virtual camera to obtain the camera perspective view corresponding to the three-dimensional point cloud under the camera perspective angle, wherein the virtual camera parameters are generated in a specified manner; According to the correspondence between the laser radar image and the camera perspective view, a conversion feature map of the laser radar image under the camera perspective view is obtained; The camera perspective image and the converted feature map are fused to obtain a fused feature map, wherein the fused feature map is used to input into a camera perspective subnetwork to assist road detection under the camera perspective view.

2. The method according to claim 1, wherein: The step of obtaining a conversion feature map of the laser radar image under the camera perspective view according to the correspondence between the laser radar image and the camera perspective view includes: Acquire a mapping diagram according to the correspondence between the laser radar image and the camera perspective image; A feature conversion map is obtained according to the mapping map and the lidar image.

3. The method according to claim 1, wherein: The step of obtaining a camera perspective view corresponding to the three-dimensional point cloud under a camera perspective angle according to the three-dimensional point cloud and the camera parameters comprises: Obtaining the first coordinate information of the image collected by the laser radar in the laser radar coordinate system; Obtaining second coordinate information corresponding to the first coordinate information in a camera coordinate system; Obtain pixel coordinate information corresponding to the second coordinate information in an image coordinate system to obtain the camera perspective image.

4. The method according to claim 3, wherein: The acquiring second coordinate information corresponding to the first coordinate information in the camera coordinate system includes: Obtaining the transformation matrix and the correction matrix of the laser radar coordinate system and the camera coordinate system; According to the transformation matrix and the correction matrix, second coordinate information corresponding to the first coordinate information in the camera coordinate system is obtained.

5. The method according to claim 3, wherein: The obtaining pixel coordinate information corresponding to the second coordinate information in the image coordinate system includes: Obtaining projection matrices of the camera coordinate system and the image coordinate system; According to the projection matrix, pixel coordinate information corresponding to the second coordinate information in an image coordinate system is obtained.

6. The method according to claim 1, wherein: The step of obtaining a laser radar image corresponding to a three-dimensional point cloud collected by the laser radar includes: Collect images through a laser radar to obtain a three-dimensional point cloud of the image; Obtain a laser radar image corresponding to the three-dimensional point cloud in a polar coordinate system.

7. A feature map fusion device, comprising: A radar image acquisition module is used to acquire a laser radar image corresponding to a three-dimensional point cloud collected by the laser radar; A camera perspective view acquisition module is used to acquire a camera perspective view corresponding to the three-dimensional point cloud under the camera perspective view according to the three-dimensional point cloud and virtual camera parameters, which includes: projecting data points of the three-dimensional point cloud onto an image plane of a virtual camera to acquire a camera perspective view corresponding to the three-dimensional point cloud under the camera perspective view, wherein the virtual camera parameters are generated in a specified manner; A conversion feature map acquisition module, used to acquire a conversion feature map of the laser radar image under the camera perspective view according to the correspondence between the laser radar image and the camera perspective view; A fusion module is used to fuse the camera perspective view and the conversion feature map to obtain a fused feature map, wherein the fused feature map is used to input into a camera perspective subnetwork to assist road detection under the camera perspective view.

8. A computer-readable storage medium storing a computer program, wherein the computer program is used to be executed by a processor to implement the feature map fusion method described in any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the feature map fusion method described in any one of claims 1-6 above.

Images