Heat map fusion method, device, terminal device and storage medium

Abstract

The present application discloses a thermal map fusion method, apparatus, terminal device and storage medium, wherein the thermal map fusion method includes: obtaining the current frame input thermal map, the previous frame output thermal map and the current frame confidence map in the vehicle coordinate system of the target vehicle; fusing the current frame input thermal map, the previous frame output thermal map and the current frame confidence map to obtain the current frame output thermal map. Based on the solution of the present application, on the basis of the current frame input thermal map and the previous frame output thermal map, the current frame confidence map is added for fusion, and the current frame confidence map can adjust the pixel weights affected by obstacles. In this way, the impact of obstacles on the fusion process can be effectively reduced, and the quality and reliability of the current frame output thermal map can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a thermal map fusion method, apparatus, terminal device, and storage medium. Background Art

[0002] In the scenario of autonomous driving, a heat map is used to determine the drivable space of the vehicle. The current frame outputs the heat map. Figure 1 This is typically achieved by fusing the output heatmap of the previous frame with the perceived input heatmap of the current frame. However, the perception information based on visual BEV (Bird's Eye View) technology is susceptible to occlusion by obstacles. The input heatmap of the current frame perceived in areas not visible to the user can be prone to jumps and low precision. In other words, due to the influence of obstacles, the vehicle's perceived input heatmap of the current frame may be inaccurate, resulting in poor quality of the fused output heatmap of the current frame. Summary of the Invention

[0003] The main purpose of this application is to provide a thermal map fusion method, device, terminal device and storage medium, aiming to solve the problem of poor quality of the thermal map output of the current frame obtained by fusion due to the influence of obstacles.

[0004] To achieve the above objectives, the present application provides a heat map fusion method, wherein the heat map is used to determine the drivable space of the target vehicle, and the heat map fusion method includes:

[0005] Obtaining a current frame input heat map, a previous frame output heat map, and a current frame confidence map of the target vehicle in the ego-vehicle coordinate system;

[0006] The current frame output heat map is obtained by fusing the current frame input heat map, the previous frame output heat map and the current frame confidence map.

[0007] Optionally, the step of obtaining the current frame confidence map of the target vehicle in the vehicle coordinate system includes:

[0008] emitting vector rays in a plurality of directions through a virtual sensor of the target vehicle and detecting obstacles in the plurality of directions;

[0009] Determining a region segmentation point based on obstacles close to the target vehicle detected by the vector ray;

[0010] Divide the area according to the area segmentation points to obtain a trusted area and an untrustworthy area;

[0011] The current frame confidence map is constructed based on the credible area and the uncredible area.

[0012] Optionally, the current frame confidence map overlaps with the current frame input heat map, and the step of fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map includes:

[0013] Obtaining a first portion of pixel points by fusing the trusted region, the current frame input heat map, and the previous frame output heat map;

[0014] A second portion of pixels is obtained by fusing the untrustworthy area, the current frame input heat map, and the previous frame output heat map;

[0015] Based on the first part of pixels and the second part of pixels, the current frame output heat map is constructed.

[0016] Optionally, the step of fusing the trusted area, the current frame input heat map, and the previous frame output heat map to obtain a first portion of pixel points includes:

[0017] If there is a first type of credible area within the credible area that overlaps with the output heat map of the previous frame, then, according to a preset update factor, pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the output heat map of the previous frame corresponding to the first type of credible area are updated to obtain a first portion of pixel points corresponding to the first type of credible area;

[0018] If there is a second type of credible area in the credible area that does not overlap with the output heat map of the previous frame, a first part of pixel points corresponding to the second type of credible area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of credible area.

[0019] Optionally, the step of fusing the untrustworthy area, the current frame input heat map, and the previous frame output heat map to obtain a second portion of pixel points and fusing the second portion of pixel points to obtain a first portion of pixel points includes:

[0020] If there is a first type of untrustworthy area in the untrustworthy area that overlaps with the output heat map of the previous frame, then according to a preset attenuation factor, the pixel values ​​of the pixel points of the output heat map of the previous frame corresponding to the first type of untrustworthy area are attenuated to obtain a second part of pixel points corresponding to the first type of untrustworthy area;

[0021] If there is a second type of untrustworthy area in the untrustworthy area that does not overlap with the output heat map of the previous frame, a second part of pixel points corresponding to the second type of untrustworthy area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of untrustworthy area.

[0022] Optionally, the step of obtaining a current frame input heat map of the target vehicle in the vehicle coordinate system includes:

[0023] Obtaining a first type of current frame input heat map in a pixel coordinate system of the target vehicle;

[0024] Upsampling the first type of current frame input heat map to obtain a second type of current frame input heat map;

[0025] Performing coordinate system conversion on the second type of current frame input heat map to obtain a third type of current frame input heat map in the vehicle coordinate system of the target vehicle;

[0026] The step of fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map includes:

[0027] A first-category current frame output heat map is obtained by fusing the third-category current frame input heat map, the previous-frame output heat map, and the current-frame confidence map;

[0028] The first type of current frame output heat map is downsampled to obtain a second type of current frame output heat map.

[0029] Optionally, the step of obtaining a previous frame output heat map of the target vehicle in the vehicle coordinate system includes:

[0030] Obtaining a current frame input heat map of the target vehicle in the ego vehicle coordinate system and a previous frame output heat map in the original coordinate system;

[0031] Calculating a change in the pose of the target vehicle based on a current frame input heat map of the target vehicle in the ego-vehicle coordinate system and a previous frame output heat map in the original coordinate system;

[0032] According to the change in the position and posture of the target vehicle, the coordinate system of the last frame output heat map in the original coordinate system is converted to obtain the last frame output heat map in the vehicle coordinate system of the target vehicle.

[0033] The present application also provides a heat map fusion device, wherein the heat map is used to determine the drivable space of the target vehicle. The heat map fusion device includes:

[0034] An acquisition module is used to obtain the current frame input heat map, the previous frame output heat map and the current frame confidence map corresponding to the target vehicle;

[0035] A fusion module is used to fuse the current frame input heat map, the previous frame output heat map and the current frame confidence map to obtain the current frame output heat map.

[0036] An embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and a heat map fusion program stored in the memory and executable on the processor. When the heat map fusion program is executed by the processor, the steps of the heat map fusion method described above are implemented.

[0037] An embodiment of the present application further provides a computer-readable storage medium, on which a heat map fusion program is stored. When the heat map fusion program is executed by a processor, the steps of the heat map fusion method described above are implemented.

[0038] The thermal map fusion method, apparatus, terminal device and storage medium proposed in the embodiment of the present application obtain the current frame input thermal map, the previous frame output thermal map and the current frame confidence map in the vehicle coordinate system of the target vehicle; and fuse the current frame input thermal map, the previous frame output thermal map and the current frame confidence map to obtain the current frame output thermal map. Based on the solution of the present application, on the basis of the current frame input thermal map and the previous frame output thermal map, the current frame confidence map is added for fusion, and the current frame confidence map can adjust the pixel weights affected by obstacles. In this way, the impact of obstacles on the fusion process can be effectively reduced, and the quality and reliability of the current frame output thermal map can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the heat map fusion device of this application belongs;

[0040] Figure 2 This is a flowchart of the first exemplary embodiment of the heat map fusion method of this application;

[0041] Figure 3 This is a flow chart of a second exemplary embodiment of the heat map fusion method of this application;

[0042] Figure 4 This is a flowchart of the third exemplary embodiment of the heat map fusion method of this application;

[0043] Figure 5 This is a flowchart of a fourth exemplary embodiment of the heat map fusion method of this application;

[0044] Figure 6 This is a flowchart of a fifth exemplary embodiment of the heat map fusion method of this application;

[0045] Figure 7 This is a flowchart of the sixth exemplary embodiment of the heat map fusion method of this application;

[0046] Figure 8 This is a flow chart of the seventh exemplary embodiment of the heat map fusion method of this application.

[0047] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0049] The main solution of the embodiment of the present application is: to obtain the current frame input heat map, the previous frame output heat map and the current frame confidence map in the vehicle coordinate system of the target vehicle; to fuse the current frame input heat map, the previous frame output heat map and the current frame confidence map to obtain the current frame output heat map. Based on the solution of the present application, on the basis of the current frame input heat map and the previous frame output heat map, the current frame confidence map is added for fusion, and the current frame confidence map can adjust the pixel weights affected by obstacles. In this way, the impact of obstacles on the fusion process can be effectively reduced, and the quality and reliability of the current frame output heat map can be improved.

[0050] Specifically, refer to Figure 1 , Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the heat map fusion device of this application belongs. The heat map fusion device can be a device independent of the terminal device and capable of performing heat map fusion, which can be hosted on the terminal device in the form of hardware or software. The terminal device can be a smart mobile terminal with data processing capabilities such as a mobile phone or tablet computer, or a fixed terminal device or server with data processing capabilities.

[0051] In this embodiment, the terminal device to which the heat map fusion apparatus belongs includes at least an output module 110 , a processor 120 , a memory 130 and a communication module 140 .

[0052] The memory 130 stores an operating system and a heat map fusion program. The heat map fusion device can obtain the current frame input heat map, the previous frame output heat map, and the current frame confidence map of the target vehicle in the ego-vehicle coordinate system; and fuse the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map and other information in the memory 130. The output module 110 can be a display screen, etc. The communication module 140 can include a WiFi module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0053] When the heat map fusion program in the memory 130 is executed by the processor, the following steps are implemented:

[0054] Obtaining a current frame input heat map, a previous frame output heat map, and a current frame confidence map of the target vehicle in the ego-vehicle coordinate system;

[0055] The current frame output heat map is obtained by fusing the current frame input heat map, the previous frame output heat map and the current frame confidence map.

[0056] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0057] emitting vector rays in a plurality of directions through a virtual sensor of the target vehicle and detecting obstacles in the plurality of directions;

[0058] Determining a region segmentation point based on obstacles close to the target vehicle detected by the vector ray;

[0059] Divide the area according to the area segmentation points to obtain a trusted area and an untrustworthy area;

[0060] The current frame confidence map is constructed based on the credible area and the uncredible area.

[0061] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0062] Obtaining a first portion of pixel points by fusing the trusted region, the current frame input heat map, and the previous frame output heat map;

[0063] A second portion of pixels is obtained by fusing the untrustworthy area, the current frame input heat map, and the previous frame output heat map;

[0064] Based on the first part of pixels and the second part of pixels, the current frame output heat map is constructed.

[0065] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0066] If there is a first type of credible area within the credible area that overlaps with the output heat map of the previous frame, then, according to a preset update factor, pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the output heat map of the previous frame corresponding to the first type of credible area are updated to obtain a first portion of pixel points corresponding to the first type of credible area;

[0067] If there is a second type of credible area in the credible area that does not overlap with the output heat map of the previous frame, a first part of pixel points corresponding to the second type of credible area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of credible area.

[0068] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0069] If there is a first type of untrustworthy area in the untrustworthy area that overlaps with the output heat map of the previous frame, then according to a preset attenuation factor, the pixel values ​​of the pixel points of the output heat map of the previous frame corresponding to the first type of untrustworthy area are attenuated to obtain a second part of pixel points corresponding to the first type of untrustworthy area;

[0070] If there is a second type of untrustworthy area in the untrustworthy area that does not overlap with the output heat map of the previous frame, a second part of pixel points corresponding to the second type of untrustworthy area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of untrustworthy area.

[0071] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0072] Obtaining a first type of current frame input heat map in a pixel coordinate system of the target vehicle;

[0073] Upsampling the first type of current frame input heat map to obtain a second type of current frame input heat map;

[0074] Performing coordinate system conversion on the second type of current frame input heat map to obtain a third type of current frame input heat map in the vehicle coordinate system of the target vehicle;

[0075] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0076] A first-category current frame output heat map is obtained by fusing the third-category current frame input heat map, the previous-frame output heat map, and the current-frame confidence map;

[0077] The first type of current frame output heat map is downsampled to obtain a second type of current frame output heat map.

[0078] Furthermore, when the heat map fusion program in the memory 130 is executed by the processor, the following steps are also implemented:

[0079] Obtaining a current frame input heat map of the target vehicle in the ego vehicle coordinate system and a previous frame output heat map in the original coordinate system;

[0080] Calculating a change in the pose of the target vehicle based on a current frame input heat map of the target vehicle in the ego-vehicle coordinate system and a previous frame output heat map in the original coordinate system;

[0081] According to the change in the position and posture of the target vehicle, the coordinate system of the last frame output heat map in the original coordinate system is converted to obtain the last frame output heat map in the vehicle coordinate system of the target vehicle.

[0082] This embodiment uses the above scheme to obtain the current frame input heat map, the previous frame output heat map, and the current frame confidence map in the vehicle coordinate system of the target vehicle; and fuses the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map. In this embodiment, the current frame confidence map is added to the current frame input heat map and the previous frame output heat map for fusion. The current frame confidence map can adjust the pixel weights affected by obstacles. In this way, the impact of obstacles on the fusion process can be effectively reduced, and the quality and reliability of the current frame output heat map can be improved.

[0083] Reference Figure 2 The first embodiment of the heat map fusion method of the present application provides a flow chart, and the heat map fusion method includes:

[0084] Step S10: Obtain the current frame input heat map, the previous frame output heat map, and the current frame confidence map of the target vehicle in the vehicle coordinate system.

[0085] Specifically, in the scenario of autonomous driving, a heat map is used to determine the vehicle’s drivable space (DrivableSpace, DS). The current frame outputs the heat map. Figure 1 It is generally obtained by fusing the output heat map of the previous frame and the perceived input heat map of the current frame. However, the perception information based on visual BEV technology is easily affected by obstacle occlusion. The current frame input heat map perceived in the visually invisible area has defects such as easy jump and low accuracy. In scenes with high occlusion, the detection reliability of 360° perception information in various directions and areas is also changing due to factors such as changes in the vehicle's moving perspective and changes in the degree of obstacle occlusion caused by other moving targets. In other words, due to the influence of obstacles, the current frame input heat map perceived by the vehicle may not be accurate, resulting in poor quality of the fused current frame output heat map.

[0086] To this end, this embodiment adds a current frame confidence map for fusion based on the current frame input heat map and the previous frame output heat map. Among them, the previous frame output heat map is obtained by fusion of the previous heat map, and the previous frame output heat map can be directly obtained; the current frame input heat map is obtained by the target vehicle's perception system perceiving the external environment; the current frame confidence map is obtained based on the perception information and virtual sensor technology, and the current frame confidence map includes credible areas and uncredible areas. More specifically, credible areas and uncredible areas are divided based on obstacles. The credible area refers to the range from the obstacle to the target vehicle; and the uncredible area refers to the range from the obstacle to the direction away from the target vehicle. It can be understood that the perception information of the credible area is accurate and credible, and the perception information of the uncredible area is uncredible, and there may be a variety of uncontrollable factors.

[0087] It can be understood that the fused input heatmap of the current frame, the output heatmap of the previous frame, and the confidence map of the current frame are all in the ego-vehicle coordinate system of the target vehicle at the current frame time. The pixels of these images correspond to coordinates in the ego-vehicle coordinate system. In other words, the positions of the pixels in the input heatmap of the current frame, the output heatmap of the previous frame, and the confidence map of the current frame can be determined.

[0088] Step S20: A current frame output heat map is obtained by fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map.

[0089] Specifically, after obtaining the current frame input heat map, the previous frame output heat map and the current frame confidence map in the vehicle coordinate system of the target vehicle, heat map fusion can be performed. The existing heat map fusion method is to directly fuse the current frame input heat map and the previous frame output heat map, while this embodiment adds the current frame confidence map on this basis. Within the credible range of the current frame confidence map, the pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the previous frame output heat map are updated. The process of pixel value update can be a weighted summation of the pixel values; within the uncredible range of the current frame confidence map, the pixel values ​​of the pixel points of the previous frame output heat map are attenuated. The process of pixel value attenuation can be to reduce the weight of the pixel value so that the pixel value is reduced. In this way, based on the fusion of the current frame input heat map, the previous frame output heat map and the current frame confidence map, the current frame output heat map can be obtained. The current frame output heat map can display the drivable area around the target vehicle, and can be used to determine the drivable space of the target vehicle in combination with the downstream processing module.

[0090] This embodiment uses the above scheme to obtain the current frame input heat map, the previous frame output heat map, and the current frame confidence map in the vehicle coordinate system of the target vehicle; and fuses the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map. In this embodiment, the current frame confidence map is added to the current frame input heat map and the previous frame output heat map for fusion. The current frame confidence map can adjust the pixel weights affected by obstacles. In this way, the impact of obstacles on the fusion process can be effectively reduced, and the quality and reliability of the current frame output heat map can be improved.

[0091] Further, refer to Figure 3 The second embodiment of the heat map fusion method of this application provides a flow chart based on the above Figure 2 In the embodiment shown, step S10, further refining the current frame confidence map of the target vehicle in the ego vehicle coordinate system, includes:

[0092] Step S11 , emitting vector rays in several directions through the virtual sensor of the target vehicle, and detecting obstacles in the several directions.

[0093] Specifically, in the specific construction method of the current frame confidence map of this embodiment, there may be obstacles around the target vehicle, which block the perception of the target vehicle. Therefore, the perception information of the current frame obtained by the target vehicle through the perception system may include obstacle information, such as the position and height of the obstacle. Vector rays are emitted in several directions through the virtual sensor of the target vehicle to detect obstacles in several directions. Among them, the virtual sensor is not a physical hardware, but a data processing algorithm; vector rays are also rays in the mathematical sense, and their direction and length can be defined.

[0094] Step S12: determining a region segmentation point based on obstacles close to the target vehicle detected by the vector ray.

[0095] Specifically, when a vector ray detects the first obstacle, that is, an obstacle close to the target vehicle, a region segmentation point is determined based on the position of the obstacle. It is understood that the region segmentation point reflects the position of the obstacle relative to the target vehicle.

[0096] Step S13: performing region division according to the region segmentation points to obtain a trusted region and an untrustworthy region.

[0097] Specifically, if a region segmentation point is determined along a vector ray, then points between the region segmentation point and the target vehicle along the vector ray are determined as points to be updated, while points in the direction away from the region segmentation point are determined as points not to be updated. It can be understood that the credibility of points to be updated is higher, while the credibility of points not to be updated is lower.

[0098] Furthermore, a trusted area can be obtained based on the points to be updated, and an untrusted area can be obtained based on the points not to be updated, thus completing the division of the trusted area and the untrusted area.

[0099] Step S14: constructing the current frame confidence map based on the credible area and the uncredible area.

[0100] Specifically, based on the division of credible areas and uncredible areas, a current frame confidence map can be further constructed, and the current frame confidence map, the current frame input heat map, and the previous frame output heat map are all of the same size.

[0101] This embodiment adopts the above scheme, specifically, by emitting vector rays in several directions through the virtual sensor of the target vehicle, and detecting obstacles in the several directions; determining region segmentation points based on the obstacles close to the target vehicle detected by the vector rays; performing region division based on the region segmentation points to obtain trusted regions and untrusted regions; and constructing the current frame confidence map based on the trusted regions and the untrusted regions.

[0102] Further, refer to Figure 4 The third embodiment of the heat map fusion method of this application provides a flow chart based on the above Figure 3 In the embodiment shown, the current frame confidence map overlaps with the current frame input heat map. Step S20 is to further refine the current frame output heat map by fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map, including:

[0103] Step S21 , fusing the trusted area, the current frame input heat map, and the previous frame output heat map to obtain a first portion of pixel points.

[0104] Specifically, the perceived information within the trusted region is more credible, and a weighted update strategy can be adopted to fuse the trusted region, the current frame input heat map, and the previous frame output heat map to obtain the first portion of pixels. It is understandable that the pixel values ​​of the first portion of pixels are higher.

[0105] Step S22: A second portion of pixels is obtained by fusing the untrustworthy area, the current frame input heat map, and the previous frame output heat map.

[0106] Specifically, the credibility of the perception information within the untrustworthy area is low, and an attenuation strategy can be adopted to fuse the second part of pixels based on the trustworthy area, the current frame input heat map, and the previous frame output heat map. It can be understood that the pixel values ​​of the second part of pixels are relatively low.

[0107] Step S23: constructing the current frame output heat map based on the first part of pixel points and the second part of pixel points.

[0108] Specifically, based on the first part of pixel points and the second part of pixel points, the current frame output heat map can be further constructed. The size of the current frame output heat map is consistent with the current frame input heat map, and the second part of pixel points in the current frame output heat map can effectively reflect the negative impact of obstacles on the accuracy of perception information, which can help downstream processing modules make better decisions.

[0109] This embodiment adopts the above scheme, specifically by fusing based on the credible area, the current frame input heat map, and the previous frame output heat map to obtain a first portion of pixel points; fusing based on the uncredible area, the current frame input heat map, and the previous frame output heat map to obtain a second portion of pixel points; and constructing the current frame output heat map based on the first portion of pixel points and the second portion of pixel points. In this embodiment, by fusing based on the credible area and the uncredible area of ​​the current frame confidence map, a first portion of pixel points with a higher credibility and a second portion of pixel points with a lower credibility can be obtained. The current frame output heat map is constructed based on the first portion of pixel points and the second portion of pixel points, which can reflect the impact of obstacles on the fusion result and improve the quality and reliability of the current frame output heat map.

[0110] Further, refer to Figure 5 The fourth embodiment of the heat map fusion method of this application provides a flow chart based on the above Figure 4 In the embodiment shown, step S21, based on the credible area, the current frame input heat map, and the previous frame output heat map, a fusion is performed to obtain a first portion of pixel points for further refinement, including:

[0111] Step S211: If there is a first type of credible area in the credible area that overlaps with the output heat map of the previous frame, the pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the previous frame output heat map corresponding to the first type of credible area are updated according to a preset update factor to obtain a first part of pixel points corresponding to the first type of credible area.

[0112] Specifically, since the position of the current frame confidence map overlaps with that of the current frame input heat map, if there is a first-type credible area in the credible area of ​​the current frame confidence map that overlaps with the output heat map of the previous frame, then the pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the previous frame output heat map corresponding to the first-type credible area can be updated according to the preset update factor.

[0113] More specifically, the update factor can be set to x, where x is generally a decimal greater than 0 and less than 1. The formula for updating the pixel value is:

[0114] U=n*x+m*(1-x)

[0115] Among them, U is the pixel value of the first part of pixels after update, n is the pixel value of the pixel of the input heat map of the current frame, and m is the pixel value of the pixel of the output heat map of the previous frame.

[0116] After the above pixel value update is performed, a first portion of pixel points corresponding to the first type of credible area can be obtained.

[0117] Step S212: If there is a second type of credible area in the credible area that does not overlap with the output heat map of the previous frame, a first portion of pixel points corresponding to the second type of credible area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of credible area.

[0118] Specifically, since the current frame confidence map overlaps with the current frame input heatmap, if there is a second type of credible region within the credible region of the current frame confidence map that does not overlap with the previous frame output heatmap, then the pixel values ​​of the pixels in the current frame input heatmap corresponding to the second type of credible region can be directly used as the pixel values ​​of the corresponding first part of pixels. In this way, the first part of pixels corresponding to the second type of credible region can be obtained.

[0119] This embodiment adopts the above scheme, specifically by updating the pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the previous frame output heat map corresponding to the first type of credible area according to a preset update factor if there is a first type of credible area in the credible area that overlaps with the previous frame output heat map, and obtaining the first part of pixel points corresponding to the first type of credible area; if there is a second type of credible area in the credible area that does not overlap with the previous frame output heat map, then based on the pixel points of the current frame input heat map corresponding to the second type of credible area, obtaining the first part of pixel points corresponding to the second type of credible area. In this embodiment, the pixel value update strategy is adopted to perform heat map fusion based on the credible area, which can obtain the first part of pixel points with higher credibility, thereby improving the quality and reliability of the current frame output heat map.

[0120] Further, refer to Figure 6 The fifth embodiment of the heat map fusion method of this application provides a flow chart based on the above Figure 4 In the embodiment shown, step S22, based on the untrustworthy area, the current frame input heat map, and the previous frame output heat map, a second portion of pixels is obtained by fusion, and the first portion of pixels is obtained by further refinement, including:

[0121] Step S221: If there is a first type of untrustworthy area in the untrustworthy area that overlaps with the output heat map of the previous frame, the pixel values ​​of the pixel points of the output heat map of the previous frame corresponding to the first type of untrustworthy area are attenuated according to a preset attenuation factor to obtain a second part of pixel points corresponding to the first type of untrustworthy area.

[0122] Specifically, since the position of the current frame confidence map overlaps with the current frame input heat map, if there is a first type of untrustworthy area in the untrustworthy area of ​​the current frame confidence map that overlaps with the output heat map of the previous frame, then the pixel value of the pixel point of the previous frame output heat map corresponding to the first type of untrustworthy area can be attenuated according to the preset attenuation factor.

[0123] More specifically, the attenuation factor can be set to y, which is generally a decimal greater than 0 and less than 1. The formula for updating the pixel value is:

[0124] V=m*y

[0125] Among them, V is the pixel value of the second part of pixels after update, and m is the pixel value of the pixel of the output heat map of the previous frame.

[0126] After the above pixel value attenuation is performed, a second portion of pixel points corresponding to the first type of untrustworthy area can be obtained.

[0127] Step S222: If there is a second type of untrustworthy area in the untrustworthy area that does not overlap with the output heat map of the previous frame, a second part of pixel points corresponding to the second type of untrustworthy area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of untrustworthy area.

[0128] Specifically, since the current frame confidence map overlaps with the current frame input heat map, if there is a second type of untrustworthy region in the untrustworthy region of the current frame confidence map that does not overlap with the previous frame output heat map, then the pixel value of the pixel point of the current frame input heat map corresponding to the second type of untrustworthy region can be directly used as the pixel value of the corresponding second part of pixels. In this way, the second part of pixels corresponding to the second type of untrustworthy region can be obtained.

[0129] This embodiment adopts the above scheme, specifically, if there is a first type of untrustworthy area in the untrustworthy area that overlaps with the output heat map of the previous frame, then according to a preset attenuation factor, the pixel values ​​of the pixel points of the output heat map of the previous frame corresponding to the first type of untrustworthy area are attenuated to obtain the second part of pixel points corresponding to the first type of untrustworthy area; if there is a second type of untrustworthy area in the untrustworthy area that does not overlap with the output heat map of the previous frame, then based on the pixel points of the current frame input heat map corresponding to the second type of untrustworthy area, the second part of pixel points corresponding to the second type of untrustworthy area are obtained. In this embodiment, the pixel value attenuation strategy is adopted to perform heat map fusion based on the untrustworthy area, which can obtain the second part of pixel points with lower credibility, thereby improving the quality and reliability of the output heat map of the current frame.

[0130] Further, refer to Figure 7 The sixth embodiment of the heat map fusion method of this application provides a flow chart based on the above Figure 2 In the embodiment shown, step S10, obtaining the current frame input heat map of the target vehicle in the ego-vehicle coordinate system and further refining it, includes:

[0131] Step S15: obtaining a first type of current frame input heat map in the pixel coordinate system of the target vehicle.

[0132] Specifically, a first type of current frame input heat map is obtained through the perception system of the target vehicle, and the first type of current frame input heat map is in the pixel coordinate system of the target vehicle.

[0133] Step S16: upsample the first type of current frame input heat map to obtain a second type of current frame input heat map.

[0134] Specifically, to ensure that the heatmap fusion process is not distorted, the first type of current frame input heatmap can be upsampled to improve the detail clarity of the image. Upsampling can be performed using interpolation algorithms such as bilinear interpolation and bicubic interpolation. After upsampling, a second type of current frame input heatmap can be obtained. It can be understood that the resolution of the second type of current frame input heatmap is higher than that of the first type of current frame input heatmap.

[0135] Step S17: performing coordinate system conversion on the second type of current frame input heat map to obtain a third type of current frame input heat map in the vehicle coordinate system of the target vehicle.

[0136] Specifically, since the second-type current frame input heatmap is in the pixel coordinate system of the target vehicle, it cannot be directly used for heatmap fusion. Therefore, the coordinate system of the second-type current frame input heatmap needs to be converted. Specifically, the coordinates of the pixel points in the second-type current frame input heatmap are converted to the coordinates in the ego vehicle coordinate system. In this way, the third-type current frame input heatmap of the target vehicle in the ego vehicle coordinate system can be obtained.

[0137] Step S20, based on the current frame input heat map, the previous frame output heat map and the current frame confidence map, a fusion is performed to obtain a current frame output heat map, which is further refined, including:

[0138] Step S24: A first-category current frame output heat map is obtained by fusing the third-category current frame input heat map, the previous-frame output heat map, and the current frame confidence map.

[0139] Specifically, the current frame confidence map includes a credible area and an untrustworthy area. Within the credible range of the current frame confidence map, the pixel values ​​of the third type of current frame input heat map and the pixel points of the previous frame output heat map are updated. The process of pixel value updating can be a weighted summation of the pixel values. Within the untrustworthy range of the current frame confidence map, the pixel values ​​of the pixel points of the previous frame output heat map are attenuated. The process of pixel value attenuation can be to reduce the weight of the pixel value to reduce the pixel value. In this way, based on the fusion of the third type of current frame input heat map, the previous frame output heat map and the current frame confidence map, the first type of current frame output heat map can be obtained.

[0140] Step S25: downsample the first type of current frame output heat map to obtain a second type of current frame output heat map.

[0141] Specifically, since step S16 obtains the second type of current frame input heat map after upsampling processing, the first type of current frame output heat map obtained by fusion still has a relatively high resolution and a relatively large amount of data, which will cause a certain amount of computational pressure on the downstream modules. To this end, the first type of current frame output heat map can be downsampled to reduce the number of pixels in the image. The downsampling can be specifically implemented based on average pooling, maximum pooling, convolution, etc. After the downsampling process, the second type of current frame output heat map can be obtained. It can be understood that the resolution of the second type of current frame output heat map is lower than that of the first type of current frame output heat map, which can reduce the computational pressure of the downstream modules.

[0142] This embodiment uses the above scheme, specifically by obtaining the first type of current frame input heat map in the pixel coordinate system of the target vehicle; upsampling the first type of current frame input heat map to obtain the second type of current frame input heat map; performing coordinate system conversion on the second type of current frame input heat map to obtain the third type of current frame input heat map in the self-vehicle coordinate system of the target vehicle; fusing the third type of current frame input heat map, the previous frame output heat map and the current frame confidence map to obtain the first type of current frame output heat map; downsampling the first type of current frame output heat map to obtain the second type of current frame output heat map. In this embodiment, the up-down sampling method is adopted to ensure that the heat map fusion process is not distorted and the data volume of the fusion result is not too large, which is conducive to improving the computing efficiency and decision quality of the downstream modules.

[0143] Further, refer to Figure 8 The seventh embodiment of the heat map fusion method of this application provides a flow chart based on the above Figure 2 In the embodiment shown, step S10, obtaining the previous frame output heat map of the target vehicle in the vehicle coordinate system and further refining it, includes:

[0144] Step S18: obtaining the current frame input heat map of the target vehicle in the vehicle coordinate system and the previous frame output heat map in the original coordinate system.

[0145] Specifically, the target vehicle's perception system can obtain the target vehicle's current frame input heatmap in the ego vehicle coordinate system. The previous frame output heatmap in the original coordinate system can be pre-stored and can be directly read to obtain the previous frame output heatmap in the original coordinate system. The original coordinate system refers to the target vehicle's ego vehicle coordinate system at the previous frame.

[0146] Step S19: Calculate the change in the position of the target vehicle based on the current frame input heat map in the ego-vehicle coordinate system and the previous frame output heat map in the original coordinate system.

[0147] Specifically, the ego-vehicle coordinate system is established based on the target vehicle's position. The target vehicle's position may change between the current frame and the previous frame, meaning the target vehicle may have moved a certain distance between the current and previous frames. The previous frame's output heatmap in the original coordinate system cannot be directly used for heatmap fusion. Therefore, a coordinate system transformation is required. Based on the current frame's input heatmap in the ego-vehicle coordinate system and the previous frame's output heatmap in the original coordinate system, the target vehicle's pose change is calculated. This pose change reflects the target vehicle's motion between the current and previous frames.

[0148] Step S110 , performing coordinate system conversion on the last frame output heat map in the original coordinate system according to the change in the position and posture of the target vehicle, to obtain the last frame output heat map in the ego-vehicle coordinate system of the target vehicle.

[0149] Specifically, after calculating the target vehicle's pose change, the coordinate system of the previous frame's output heatmap in the original coordinate system can be transformed based on the pose change. The coordinates of the pixel points in the previous frame's output heatmap in the original coordinate system are converted to the coordinates in the current frame's ego vehicle coordinate system. In this way, the target vehicle's previous frame's output heatmap in the ego vehicle coordinate system can be obtained.

[0150] This embodiment uses the above scheme, specifically by obtaining the current frame input heat map of the target vehicle in the ego vehicle coordinate system and the previous frame output heat map in the original coordinate system; based on the current frame input heat map of the target vehicle in the ego vehicle coordinate system and the previous frame output heat map in the original coordinate system, calculating the target vehicle's posture change; according to the target vehicle's posture change, the previous frame output heat map in the original coordinate system is subjected to a coordinate system conversion to obtain the previous frame output heat map in the ego vehicle coordinate system of the target vehicle. In this embodiment, the posture change is used to convert the previous frame output heat map from the original coordinate system to the target vehicle's ego vehicle coordinate system to meet the basic requirements of heat map fusion.

[0151] In addition, an embodiment of the present application further provides a heat map fusion device, wherein the heat map is used to determine the drivable space of the target vehicle, and the heat map fusion device includes:

[0152] An acquisition module is used to obtain the current frame input heat map, the previous frame output heat map and the current frame confidence map corresponding to the target vehicle;

[0153] A fusion module is used to fuse the current frame input heat map, the previous frame output heat map and the current frame confidence map to obtain the current frame output heat map.

[0154] For the principle and implementation process of heat map fusion in this embodiment, please refer to the above embodiments and will not be repeated here.

[0155] In addition, an embodiment of the present application also proposes a terminal device, which includes a memory, a processor, and a thermal map fusion program stored on the memory and runnable on the processor. When the thermal map fusion program is executed by the processor, the steps of the thermal map fusion method described above are implemented.

[0156] Since this heat map fusion program adopts all the technical solutions of all the aforementioned embodiments when executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0157] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which a heat map fusion program is stored. When the heat map fusion program is executed by a processor, the steps of the heat map fusion method described above are implemented.

[0158] Since this heat map fusion program adopts all the technical solutions of all the aforementioned embodiments when executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0159] Compared with the existing technology, the thermal map fusion method, device, terminal device and storage medium proposed in the embodiment of the present application obtain the current frame input thermal map, the previous frame output thermal map and the current frame confidence map in the vehicle coordinate system of the target vehicle; and fuse the current frame input thermal map, the previous frame output thermal map and the current frame confidence map to obtain the current frame output thermal map. Based on the solution of the present application, on the basis of the current frame input thermal map and the previous frame output thermal map, the current frame confidence map is added for fusion, and the current frame confidence map can adjust the pixel weights affected by obstacles. In this way, the impact of obstacles on the fusion process can be effectively reduced, and the quality and reliability of the current frame output thermal map can be improved.

[0160] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0161] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the method of each embodiment of the present application.

[0163] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A heat map fusion method, characterized in that: The heat map is used to determine the drivable space of the target vehicle, and the heat map fusion method includes: Obtaining a current frame input heat map, a previous frame output heat map, and a current frame confidence map of the target vehicle in the ego-vehicle coordinate system; Fusion is performed based on the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain a current frame output heat map; The step of obtaining the current frame confidence map of the target vehicle in the vehicle coordinate system includes: Vector rays are emitted in several directions by the virtual sensor of the target vehicle, and obstacles in the several directions are detected; region segmentation points are determined based on the obstacles close to the target vehicle detected by the vector rays; regions are divided according to the region segmentation points to obtain credible regions and uncredible regions; and a confidence map of the current frame is constructed based on the credible regions and the uncredible regions; The current frame confidence map overlaps with the current frame input heat map, and the step of fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map includes: Based on the trusted area, the current frame input heat map, and the previous frame output heat map, a first part of pixel points is obtained by fusion; based on the untrustworthy area, the current frame input heat map, and the previous frame output heat map, a second part of pixel points is obtained by fusion; based on the first part of pixel points and the second part of pixel points, the current frame output heat map is constructed.

2. The heat map fusion method according to claim 1, characterized in that: The step of fusing the trusted area, the current frame input heat map, and the previous frame output heat map to obtain a first portion of pixel points includes: If there is a first type of credible area within the credible area that overlaps with the output heat map of the previous frame, then, according to a preset update factor, pixel values ​​of the pixel points of the current frame input heat map and the pixel points of the output heat map of the previous frame corresponding to the first type of credible area are updated to obtain a first portion of pixel points corresponding to the first type of credible area; If there is a second type of credible area in the credible area that does not overlap with the output heat map of the previous frame, a first part of pixel points corresponding to the second type of credible area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of credible area.

3. The heat map fusion method according to claim 1, characterized in that: The step of fusing the untrustworthy area, the current frame input heat map, and the previous frame output heat map to obtain the second part of pixel points and fusing them to obtain the first part of pixel points includes: If there is a first type of untrustworthy area in the untrustworthy area that overlaps with the output heat map of the previous frame, then according to a preset attenuation factor, the pixel values ​​of the pixel points of the output heat map of the previous frame corresponding to the first type of untrustworthy area are attenuated to obtain a second part of pixel points corresponding to the first type of untrustworthy area; If there is a second type of untrustworthy area in the untrustworthy area that does not overlap with the output heat map of the previous frame, a second part of pixel points corresponding to the second type of untrustworthy area is obtained based on the pixel points of the current frame input heat map corresponding to the second type of untrustworthy area.

4. The heat map fusion method according to claim 1, wherein: The step of obtaining the current frame input heat map of the target vehicle in the vehicle coordinate system includes: Obtaining a first type of current frame input heat map in a pixel coordinate system of the target vehicle; Upsampling the first type of current frame input heat map to obtain a second type of current frame input heat map; Performing coordinate system conversion on the second type of current frame input heat map to obtain a third type of current frame input heat map in the vehicle coordinate system of the target vehicle; The step of fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map includes: A first-category current frame output heat map is obtained by fusing the third-category current frame input heat map, the previous-frame output heat map, and the current-frame confidence map; The first type of current frame output heat map is downsampled to obtain a second type of current frame output heat map.

5. The heat map fusion method according to claim 1, characterized in that: The step of obtaining the last frame output heat map of the target vehicle in the vehicle coordinate system includes: Obtaining a current frame input heat map of the target vehicle in the ego vehicle coordinate system and a previous frame output heat map in the original coordinate system; Calculating a change in the pose of the target vehicle based on a current frame input heat map of the target vehicle in the ego-vehicle coordinate system and a previous frame output heat map in the original coordinate system; According to the change in the position and posture of the target vehicle, the coordinate system of the last frame output heat map in the original coordinate system is converted to obtain the last frame output heat map in the vehicle coordinate system of the target vehicle.

6. A thermal map fusion device, characterized in that: The thermal map is used to determine the drivable space of the target vehicle, and the thermal map fusion device includes: An acquisition module is used to obtain the current frame input heat map, the previous frame output heat map and the current frame confidence map corresponding to the target vehicle; A fusion module, configured to fuse the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain a current frame output heat map; The acquisition module is further configured to emit vector rays in a plurality of directions through the virtual sensor of the target vehicle and detect obstacles in the plurality of directions; Determining a region segmentation point based on obstacles close to the target vehicle detected by the vector ray; Divide the area according to the area segmentation points to obtain a trusted area and an untrustworthy area; Constructing the current frame confidence map based on the credible area and the uncredible area; The current frame confidence map overlaps with the current frame input heat map, and the step of fusing the current frame input heat map, the previous frame output heat map, and the current frame confidence map to obtain the current frame output heat map includes: Based on the trusted area, the current frame input heat map, and the previous frame output heat map, a first part of pixel points is obtained by fusion; based on the untrustworthy area, the current frame input heat map, and the previous frame output heat map, a second part of pixel points is obtained by fusion; based on the first part of pixel points and the second part of pixel points, the current frame output heat map is constructed.

7. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a heat map fusion program stored in the memory and executable on the processor. When the heat map fusion program is executed by the processor, the steps of the heat map fusion method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a heat map fusion program, which, when executed by a processor, implements the steps of the heat map fusion method according to any one of claims 1 to 5.

Images