Roadside thunder vision fusion perception method and device combined with high-precision map

Abstract

The invention discloses a roadside thunder-vision fusion perception method and device combined with a high-precision map, and relates to the technical field of information, and the method comprises the steps: obtaining image data shot by a camera in real time; performing semantic segmentation on the image data to obtain road elements in the image data; determining prior information of the road elements in a map; performing target detection and target tracking on the image data in sequence to obtain an image detection result; fusing the image detection result with a radar detection result to obtain a fused sensing result; and based on the prior information, respectively correcting fusion position coordinates and fusion sizes of the traffic participants in the fusion perception result to obtain a corrected fusion perception result. According to the invention, the position and size sensing precision of the traffic participant can be improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a roadside radar-visual fusion perception method and device combined with a high-precision map. Background Art

[0002] Intelligent connected vehicles are closely integrated with transportation, energy, and cities, carrying important missions such as the strategic transformation of the national economy. In order to promote the high-quality and rapid development of intelligent connected vehicles, my country has proposed a development route of vehicle-road-cloud integration, among which roadside perception can provide detailed information about traffic participants, which is an important part of the development of China's intelligent connected vehicles.

[0003] At present, cameras and millimeter-wave radars are usually used to perceive traffic participants when performing roadside perception. However, in existing solutions, the position and size perception accuracy of traffic participants is heavily dependent on the calibration accuracy of the sensors and is limited by the sensor properties of cameras and millimeter-wave radars, resulting in low position and size perception accuracy of traffic participants, which is difficult to meet the use requirements of the vehicle side. Summary of the invention

[0004] The present invention provides a roadside radar-visual fusion perception method and device combined with a high-precision map, which is mainly capable of improving the position and size perception accuracy of traffic participants.

[0005] According to a first aspect of an embodiment of the present invention, a roadside radar-visual fusion perception method combined with a high-precision map is provided, comprising:

[0006] Get the image data captured by the camera in real time;

[0007] Performing semantic segmentation on the image data to obtain road elements in the image data;

[0008] Determining prior information of the road element in the map;

[0009] Performing target detection and target tracking on the image data in sequence to obtain an image detection result;

[0010] Fusing the image detection result with the radar detection result to obtain a fused perception result;

[0011] Based on the prior information, the fused position coordinates and fused sizes of the traffic participants in the fused perception result are corrected respectively to obtain a corrected fused perception result.

[0012] According to a second aspect of an embodiment of the present invention, a roadside radar-visual fusion perception device combined with a high-precision map is provided, comprising:

[0013] An acquisition unit, used to acquire image data captured by a camera in real time;

[0014] A segmentation unit, used for performing semantic segmentation on the image data to obtain road elements in the image data;

[0015] A determination unit, used to determine the prior information of the road element in the map;

[0016] A detection unit, used to perform target detection and target tracking on the image data in sequence to obtain an image detection result;

[0017] A fusion unit, used to fuse the image detection result with the radar detection result to obtain a fusion perception result;

[0018] The correction unit is used to correct the fusion position coordinates and fusion size of the traffic participants in the fusion perception result based on the prior information to obtain a corrected fusion perception result.

[0019] According to a third aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0020] Get the image data captured by the camera in real time;

[0021] Performing semantic segmentation on the image data to obtain road elements in the image data;

[0022] Determining prior information of the road element in the map;

[0023] Performing target detection and target tracking on the image data in sequence to obtain an image detection result;

[0024] Fusing the image detection result with the radar detection result to obtain a fused perception result;

[0025] Based on the prior information, the fused position coordinates and fused sizes of the traffic participants in the fused perception result are corrected respectively to obtain a corrected fused perception result.

[0026] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0027] Get the image data captured by the camera in real time;

[0028] Performing semantic segmentation on the image data to obtain road elements in the image data;

[0029] Determining prior information of the road element in the map;

[0030] Performing target detection and target tracking on the image data in sequence to obtain an image detection result;

[0031] Fusing the image detection result with the radar detection result to obtain a fused perception result;

[0032] Based on the prior information, the fused position coordinates and fused sizes of the traffic participants in the fused perception result are corrected respectively to obtain a corrected fused perception result.

[0033] The present invention provides a roadside radar-visual fusion perception method and device combined with a high-precision map. Compared with the prior art, it can obtain image data captured by a camera in real time, and perform semantic segmentation on the image data to obtain the road elements in the image data and determine the prior information of the road elements in the map. At the same time, the image data is sequentially subjected to target detection and target tracking to obtain image detection results, and the image detection results are fused with the radar detection results to obtain fused perception results. Finally, based on the prior information, the fused position coordinates and fused size of the traffic participants in the fused perception results are respectively corrected to obtain the corrected fused perception results. It can be seen that the present invention can improve the position and size perception accuracy of traffic participants by introducing a high-precision map, obtaining the prior information of road elements in the map, and correcting the position and size of traffic participants based on the prior information.

[0034] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of the process of a roadside radar-visual fusion perception method combined with a high-precision map provided by an embodiment of the present invention is shown;

[0037] Figure 2 The overall process diagram provided by the embodiment of the present invention is shown;

[0038] Figure 3 A schematic diagram of a post-fusion process provided by an embodiment of the present invention is shown;

[0039] Figure 4 A schematic diagram showing the positions of traffic participants and lane lines provided by an embodiment of the present invention is shown;

[0040] Figure 5 A schematic diagram of the structure of a roadside radar and vision fusion perception device combined with a high-precision map provided by an embodiment of the present invention is shown;

[0041] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0044] In the existing solutions, the position and size perception accuracy of traffic participants is heavily dependent on the calibration accuracy of sensors and is limited by the sensor properties of cameras and millimeter-wave radars, resulting in low position and size perception accuracy of traffic participants, which makes it difficult to meet the usage requirements of the vehicle side.

[0045] In order to overcome the above defects, the embodiment of the present invention provides a roadside radar-visual fusion perception method combined with a high-precision map, such as Figure 1 As shown, the method includes:

[0046] Step 101: Acquire image data captured by a camera in real time.

[0047] The image data is an image of a road area taken by a camera.

[0048] The embodiment of the present invention is mainly applicable to the scenario of correcting the perceived position and size of a traffic participant. The execution subject of the embodiment of the present invention is a device or equipment that can correct the position and size of a traffic participant.

[0049] The overall process of correcting the position and size of a traffic participant in the embodiment of the present invention is as follows: Figure 2 As shown in the figure, the entire perception process uses cameras and millimeter-wave radars. During the specific perception, the camera will be used to capture the image data of the road in real time, and then the acquired image data will be decoded and preprocessed. The preprocessing operations specifically include denoising, scaling and other operations.

[0050] Step 102: semantically segment the image data to obtain road elements in the image data.

[0051] Among them, road elements include lane lines, road marking lines, etc.

[0052] According to the embodiment of the present invention, after decoding and preprocessing operations, semantic segmentation is performed on the image data, road elements in the image data are extracted, and pixel coordinates of each endpoint on the road element in the image data are determined.

[0053] Step 103: Determine the prior information of the road element in the map.

[0054] The prior information includes the position coordinates of each endpoint of the road element in the map and the actual size of the road element.

[0055] For the embodiment of the present invention, after extracting the road elements from the image data, it is necessary to determine the prior information of the road elements in the high-precision map. For this process, step 103 specifically includes: based on the image calibration matrix, coordinate conversion is performed on the pixel coordinates of the road elements obtained by segmentation to obtain the position coordinates of the road elements in the world coordinate system; according to the position coordinates corresponding to the road elements obtained by segmentation and the position coordinates corresponding to the road elements in the map, the road elements obtained by segmentation are matched with the road elements in the map, and according to the matching results, the prior information of the road elements obtained by segmentation in the map is determined.

[0056] Furthermore, according to the position coordinates corresponding to the road elements obtained by segmentation and the position coordinates corresponding to the road elements in the map, the road elements obtained by segmentation are matched with the road elements in the map, and according to the matching results, the prior information of the road elements obtained by segmentation in the map is determined, including: calculating the Euclidean distance between the road elements obtained by segmentation and the road elements in the map according to the position coordinates corresponding to the road elements obtained by segmentation and the position coordinates corresponding to the road elements in the map; determining the target road element in the map that matches the road elements obtained by segmentation based on the Euclidean distance; and determining the prior information of the road elements obtained by segmentation in the map according to the position coordinates and size of the target road elements in the map.

[0057] Specifically, the Euclidean distances between the road elements obtained by segmentation and the road elements in the map are calculated respectively, and the minimum Euclidean distance is determined from the various Euclidean distances. The road element corresponding to the minimum Euclidean distance in the map is determined as the target road element that matches the road element obtained by segmentation. Then, the prior information of the road element obtained by segmentation in the map is determined based on the position coordinates and size of the target road element in the map.

[0058] Step 104: perform target detection and target tracking on the image data in sequence to obtain an image detection result.

[0059] For the embodiment of the present invention, it is necessary not only to perform semantic segmentation on the image data, but also to perform target detection and target tracking on the image data. For this process, step 104 specifically includes: performing target detection on the image data to obtain the traffic participants in the image data, and the categories, frame pixel coordinates and sizes corresponding to the traffic participants; determining the position coordinates of the traffic participants in the world coordinate system based on the frame pixel coordinates and the image calibration matrix; performing target tracking on the traffic participants to determine the image identification and speed corresponding to the traffic participants; determining the attribute information corresponding to the traffic participants based on the categories, position coordinates, sizes, image identification and speed corresponding to the traffic participants; and determining the image detection result based on the attribute information corresponding to the traffic participants.

[0060] The frame pixel coordinates include the pixel coordinates of the upper left corner of the frame, the pixel coordinates of the lower right corner, and the pixel coordinates of the center point of the lower edge of the frame. The categories of traffic participants include vehicles, pedestrians, etc.

[0061] It should be noted that when tracking a target, if there are any missing frames, they can be supplemented by prediction.

[0062] Step 105: Fuse the image detection result with the radar detection result to obtain a fused perception result.

[0063] According to the embodiment of the present invention, after obtaining the image detection results and radar detection results of the traffic participants, it is necessary to fuse the image detection results and the radar detection results. For this process, Figure 3As shown, the method includes: according to the attribute information corresponding to the traffic participants in the image detection results and the attribute information corresponding to the traffic participants in the radar detection results, one-to-one correspondence is made between the traffic participants in the image detection results and the traffic participants in the radar detection results to obtain the corresponding traffic participants; for any one of the corresponding traffic participants, the attribute information of the any one traffic participant in the image detection results is fused with the attribute information of the any one traffic participant in the radar detection results to obtain the fused perception result corresponding to the any one traffic participant.

[0064] Furthermore, when making a one-to-one correspondence between traffic participants in the image detection results and traffic participants in the radar detection results, the method includes: calculating the similarity between the traffic participants in the image detection results and the traffic participants in the radar detection results based on the attribute information corresponding to the traffic participants in the image detection results and the attribute information corresponding to the traffic participants in the radar detection results; and making a one-to-one correspondence between the traffic participants in the image detection results and the traffic participants in the radar detection results based on the similarity to obtain the corresponding traffic participants.

[0065] Specifically, when performing fusion, the radar and camera need to be time synchronized. After the image detection results and radar detection results come in, the two most recent frames of data are selected and the two frames of data are associated and matched, that is, similarity calculation is performed. If the similarity between a traffic participant in the image detection result and a traffic participant in the radar detection result is greater than a preset threshold, it means that the two traffic participants are the same traffic participant; on the contrary, if the similarity between a traffic participant in the image detection result and a traffic participant in the radar detection result is less than or equal to the preset threshold, it means that the two traffic participants are not the same traffic participant.

[0066] Furthermore, when fusing the attribute information of any traffic participant in the image detection result with its attribute information in the radar detection result, the method further includes: when the attribute information is an identifier, determining the fused identifier corresponding to the any traffic participant according to the image identifier of the any traffic participant in the image detection result and the radar identifier in the radar detection result; when the attribute information is speed, weighting the speed of the any traffic participant in the image detection result and the speed in the radar detection result to determine the fused speed corresponding to the any traffic participant; when the attribute information is position coordinates, determining the fused speed corresponding to the any traffic participant according to the position coordinates of the any traffic participant in the image detection result. The fused position coordinates corresponding to any one of the traffic participants are determined based on the position coordinates in the detection result or the position coordinates in the radar detection result; when the attribute information is a category, the fused category corresponding to any one of the traffic participants is determined based on the category of any one of the traffic participants in the image detection result; when the attribute information is a size, the fused size corresponding to any one of the traffic participants is determined based on the size of any one of the traffic participants in the image detection result or the size of the radar detection result; the fused perception result corresponding to any one of the traffic participants is determined based on the fusion identifier, the fusion speed, the fusion position coordinates, the fusion category and the fusion size.

[0067] Specifically, when performing frame fusion, the final fusion identification includes image identification and radar identification; the final fusion category trusts the detection result of the image; the final fusion speed trusts the radar more, so the weight of the radar will be higher; the final fusion distance can be determined according to the distance between the sensor and the detection object, the image detection result is more trusted for the near one, and the radar detection result is more trusted for the far one. The longitudinal distance will trust the radar detection result more, and the lateral distance will trust the image detection result more.

[0068] It should be noted that the attribute information in the embodiment of the present invention includes not only the image identification, radar identification, speed, category, position coordinates and size, but also the heading angle and the like.

[0069] Step 106: Based on the prior information, the fused position coordinates and fused sizes of the traffic participants in the fused perception result are respectively corrected to obtain a corrected fused perception result.

[0070] For the embodiment of the present invention, after obtaining the prior information of the road element, the position and size of the traffic participant can be corrected based on the prior information. For this process, the method includes: determining the position coordinates of each endpoint on the road element in the map according to the prior information; determining the pixel coordinates of each endpoint on the road element in the image data; based on the position coordinates of each endpoint on the road element in the map and the pixel coordinates in the image data, as well as the border pixel coordinates corresponding to the traffic participant, respectively correcting the fused position coordinates and fused size of the traffic participant to obtain a corrected fusion perception result.

[0071] Among them, according to the semantic segmentation result, the pixel coordinates of each endpoint on the road element in the image data can be determined.

[0072] Furthermore, when the endpoints on the road element include a first endpoint, a second endpoint, a third endpoint, and a fourth endpoint, based on the position coordinates of the endpoints on the road element in the map and the pixel coordinates in the image data, as well as the border pixel coordinates corresponding to the traffic participant, the fusion position coordinates and fusion size of the traffic participant are respectively corrected to obtain a corrected fusion perception result, including: determining the pixel coordinates, border pixel width, and border pixel height of the center point of the lower edge of the border based on the border pixel coordinates corresponding to the traffic participant; calculating the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, and the vertical pixel difference between the second endpoint and the fourth endpoint based on the pixel coordinates corresponding to the first endpoint, the second endpoint, the third endpoint, and the fourth endpoint, respectively, and the pixel coordinates of the center point of the lower edge of the border; calculating the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the third endpoint, the horizontal pixel difference and the vertical pixel difference between the first endpoint and the second endpoint, and the vertical pixel difference between the second endpoint and the fourth endpoint based on the first endpoint, the second endpoint, the third endpoint, and the fourth endpoint, respectively. The distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint are determined according to the position coordinates corresponding to the second endpoint, the third endpoint, and the fourth endpoint respectively; the fusion position coordinates and the fusion size are corrected respectively according to the pixel coordinates of the center point of the lower edge of the border, the pixel width of the border, the pixel height of the border, the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, the vertical pixel difference between the second endpoint and the fourth endpoint, the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint to obtain the corrected position coordinates and the corrected size; the corrected fusion perception result is determined according to the corrected position coordinates and the corrected size.

[0073] Further, according to the pixel coordinates of the center point of the lower edge of the border, the pixel width of the border, the pixel height of the border, the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, the vertical pixel difference between the second endpoint and the fourth endpoint, the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint, the fusion position coordinates and the fusion size are respectively corrected to obtain the corrected position coordinates and the corrected size, including: according to the horizontal pixel difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, and the vertical pixel difference between the first endpoint and the fourth endpoint According to the distance between the first and third endpoints, the horizontal correction distance is calculated; according to the vertical coordinate difference between the first endpoint and the center point of the lower edge of the frame, the horizontal pixel difference between the first endpoint and the second endpoint, and the distance between the first endpoint and the second endpoint, the vertical correction distance is calculated; based on the horizontal correction distance and the vertical correction distance, the fusion position coordinates are corrected to obtain the corrected position coordinates; according to the frame pixel width, the horizontal pixel difference between the first endpoint and the third endpoint, and the distance between the first endpoint and the third endpoint, the correction width is calculated; according to the frame pixel height, the vertical pixel difference between the second endpoint and the fourth endpoint, and the distance between the second endpoint and the fourth endpoint, the correction length is calculated; according to the correction width and the correction length, the fusion size is corrected to obtain the corrected size.

[0074] Specifically, Figure 4 As shown, V represents the vehicle, the first endpoint, the second endpoint, the third endpoint and the fourth endpoint on the road element are point A, point B, point C and point E respectively, and the center point of the lower edge of the border is point L. Based on the prior information of the road element, the position coordinates of each endpoint on the road element in the map can be determined. At the same time, based on the semantic segmentation results, the pixel coordinates of each endpoint on the road element in the image data can be determined. Therefore, the position and size of the traffic participants can be corrected according to the position coordinates of each endpoint on the road element in the map and the pixel coordinates in the image data. Among them, the specific formula for position correction is as follows:

[0075] LA horizontal distance = (AL horizontal pixel difference / AC horizontal pixel difference) * AC distance

[0076] LA vertical distance = (AL vertical pixel difference / AB horizontal pixel difference) * AB distance

[0077] Among them, the lateral distance of LA is the lateral correction distance, and the longitudinal distance of LA is the longitudinal correction distance.

[0078] Furthermore, the specific formula for size correction is as follows:

[0079] Correction width = vehicle border pixel width / AC horizontal pixel difference * AC distance

[0080] Corrected length = vehicle border pixel height / EB longitudinal pixel difference * EB distance

[0081] Therefore, the position and size of the traffic participant can be corrected according to the above distance correction formula and size correction formula, thereby improving the perception accuracy of the position and size of the traffic participant.

[0082] An embodiment of the present invention provides a roadside radar-visual fusion perception method combined with a high-precision map. By introducing a high-precision map, the prior information of road elements in the map is obtained, and based on the prior information, the position and size of traffic participants are corrected, thereby improving the perception accuracy of the position and size of traffic participants.

[0083] Further, as Figure 1 In the specific implementation of the present invention, an embodiment of the present invention provides a roadside radar and visual fusion perception device combined with a high-precision map, such as Figure 5 As shown, the device includes: an acquisition unit 31, a segmentation unit 32, a determination unit 33, a detection unit 34, a fusion unit 35 and a correction unit 36.

[0084] The acquisition unit 31 may be used to acquire image data captured by a camera in real time.

[0085] The segmentation unit 32 may be used to perform semantic segmentation on the image data to obtain road elements in the image data.

[0086] The determining unit 33 may be used to determine the prior information of the road element in the map.

[0087] The detection unit 34 may be used to perform target detection and target tracking on the image data in sequence to obtain an image detection result.

[0088] The fusion unit 35 can be used to fuse the image detection result with the radar detection result to obtain a fused perception result.

[0089] The correction unit 36 ​​may be used to correct the fused position coordinates and fused size of the traffic participant in the fused perception result based on the prior information to obtain a corrected fused perception result.

[0090] In a specific application scenario, the determination unit 33 includes: a conversion module and a matching module.

[0091] The conversion module may be used to perform coordinate conversion on the pixel coordinates of the road elements obtained by segmentation based on an image calibration matrix, so as to obtain the position coordinates of the road elements in a world coordinate system.

[0092] The matching module can be used to match the road elements obtained by segmentation with the road elements in the map based on the position coordinates corresponding to the road elements obtained by segmentation and the position coordinates corresponding to the road elements in the map, and determine the prior information of the road elements obtained by segmentation in the map based on the matching results.

[0093] Furthermore, the matching module can be specifically used to calculate the Euclidean distance between the road element obtained by segmentation and the road element in the map based on the position coordinates corresponding to the road element obtained by segmentation and the position coordinates corresponding to the road element in the map; based on the Euclidean distance, determine the target road element in the map that matches the road element obtained by segmentation; and determine the prior information of the road element obtained by segmentation in the map based on the position coordinates and size of the target road element in the map.

[0094] In a specific application scenario, the detection unit 34 can be specifically used to perform target detection on the image data to obtain the traffic participants in the image data, as well as the categories, border pixel coordinates and sizes corresponding to the traffic participants; determine the position coordinates of the traffic participants in the world coordinate system based on the border pixel coordinates and the image calibration matrix; track the traffic participants to determine the image identification and speed corresponding to the traffic participants; determine the attribute information corresponding to the traffic participants based on the categories, position coordinates, sizes, image identification and speed corresponding to the traffic participants; and determine the image detection results based on the attribute information corresponding to the traffic participants.

[0095] In a specific application scenario, the fusion unit 35 can be specifically used to make a one-to-one correspondence between the traffic participants in the image detection results and the traffic participants in the radar detection results according to the attribute information corresponding to the traffic participants in the image detection results and the attribute information corresponding to the traffic participants in the radar detection results, so as to obtain the corresponding traffic participants; for any one of the corresponding traffic participants, the attribute information of any one of the traffic participants in the image detection results is fused with the attribute information of any one of the traffic participants in the radar detection results, so as to obtain a fused perception result corresponding to any one of the traffic participants.

[0096] In a specific application scenario, the correction unit 36 ​​includes: a determination module and a correction module.

[0097] The determination module may be used to determine the position coordinates of each endpoint of the road element in the map according to the prior information.

[0098] The determination module may also be used to determine the pixel coordinates of each endpoint on the road element in the image data.

[0099] The correction module can be used to correct the fusion position coordinates and fusion size of the traffic participants based on the position coordinates of each endpoint on the road element in the map and the pixel coordinates in the image data, as well as the border pixel coordinates corresponding to the traffic participants, to obtain a corrected fusion perception result.

[0100] Furthermore, when the endpoints on the road element include a first endpoint, a second endpoint, a third endpoint and a fourth endpoint, the correction module includes a determination submodule, a calculation submodule and a correction submodule.

[0101] The determination submodule may be used to determine the pixel coordinates of the center point of the lower edge of the border, the border pixel width, and the border pixel height based on the border pixel coordinates corresponding to the traffic participant.

[0102] The calculation submodule can be used to calculate the horizontal pixel difference and vertical coordinate difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, and the vertical pixel difference between the second endpoint and the fourth endpoint based on the pixel coordinates corresponding to the first endpoint, the second endpoint, the third endpoint and the fourth endpoint, respectively, and the pixel coordinates of the center point of the lower edge of the border.

[0103] The determination submodule can also be used to determine the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint based on the position coordinates corresponding to the first endpoint, the second endpoint, the third endpoint and the fourth endpoint respectively.

[0104] The correction submodule can be used to correct the fusion position coordinates and the fusion size according to the pixel coordinates of the center point of the lower edge of the border, the pixel width of the border, the pixel height of the border, the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, the vertical pixel difference between the second endpoint and the fourth endpoint, the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint, to obtain the corrected position coordinates and the corrected size.

[0105] The determination submodule may also be used to determine the corrected fusion perception result according to the corrected position coordinates and the corrected size.

[0106] Further, the correction submodule can be specifically used to calculate a horizontal correction distance according to the horizontal pixel difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the third endpoint, and the distance between the first endpoint and the third endpoint; calculate a vertical correction distance according to the vertical coordinate difference between the first endpoint and the center point of the lower edge of the border, the horizontal pixel difference between the first endpoint and the second endpoint, and the distance between the first endpoint and the second endpoint; based on the horizontal correction distance and the vertical correction distance, correct the fusion position coordinates to obtain the corrected position coordinates; calculate a correction width according to the border pixel width, the horizontal pixel difference between the first endpoint and the third endpoint, and the distance between the first endpoint and the third endpoint; calculate a correction length according to the border pixel height, the vertical pixel difference between the second endpoint and the fourth endpoint, and the distance between the second endpoint and the fourth endpoint; correct the fusion size according to the correction width and the correction length to obtain the corrected size.

[0107] It should be noted that for other corresponding descriptions of the functional modules involved in the roadside radar and vision fusion perception device combined with a high-precision map provided in an embodiment of the present invention, please refer to Figure 1 The corresponding description of the method shown will not be repeated here.

[0108] Based on the above Figure 1The method shown, accordingly, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: acquiring image data taken by a camera in real time; performing semantic segmentation on the image data to obtain road elements in the image data; determining prior information of the road elements in the map; performing target detection and target tracking on the image data in sequence to obtain image detection results; fusing the image detection results with radar detection results to obtain fused perception results; based on the prior information, respectively correcting the fused position coordinates and fused sizes of the traffic participants in the fused perception results to obtain a corrected fused perception result.

[0109] Based on the above Figure 1 The method shown and Figure 5 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of an electronic device, such as Figure 6 As shown, the electronic device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 implements the following steps when executing the program: acquiring image data captured by a camera in real time; performing semantic segmentation on the image data to obtain road elements in the image data; determining prior information of the road elements in a map; performing target detection and target tracking on the image data in sequence to obtain image detection results; fusing the image detection results with radar detection results to obtain fused perception results; based on the prior information, respectively correcting the fused position coordinates and fused sizes of the traffic participants in the fused perception results to obtain corrected fused perception results.

[0110] The embodiment of the present invention introduces a high-precision map to obtain prior information of road elements in the map, and based on the prior information, corrects the position and size of traffic participants, thereby improving the perception accuracy of the position and size of traffic participants.

[0111] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0112] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A roadside radar and visual fusion perception method combined with a high-precision map, characterized in that: include: Get the image data captured by the camera in real time; Performing semantic segmentation on the image data to obtain road elements in the image data; Determining prior information of the road element in the map; Performing target detection and target tracking on the image data in sequence to obtain an image detection result; Fusing the image detection result with the radar detection result to obtain a fused perception result; Based on the prior information, the fused position coordinates and fused sizes of the traffic participants in the fused perception result are corrected respectively to obtain a corrected fused perception result.

2. The method according to claim 1, characterized in that The determining of the prior information of the road element in the map includes: Based on the image calibration matrix, coordinate transformation is performed on the pixel coordinates of the road element obtained by segmentation to obtain the position coordinates of the road element in the world coordinate system; According to the position coordinates corresponding to the road elements obtained by segmentation and the position coordinates corresponding to the road elements in the map, the road elements obtained by segmentation are matched with the road elements in the map, and according to the matching results, the prior information of the road elements obtained by segmentation in the map is determined.

3. The method according to claim 2, characterized in that According to the position coordinates corresponding to the road elements obtained by segmentation and the position coordinates corresponding to the road elements in the map, the road elements obtained by segmentation are matched with the road elements in the map, and according to the matching results, the prior information of the road elements obtained by segmentation in the map is determined, including: Calculating the Euclidean distance between the road element obtained by segmentation and the road element in the map according to the position coordinates corresponding to the road element obtained by segmentation and the position coordinates corresponding to the road element in the map; Based on the Euclidean distance, determining a target road element in the map that matches the road element obtained by segmentation; According to the position coordinates and size of the target road element in the map, the prior information of the road element in the map obtained by segmentation is determined.

4. The method according to claim 1, characterized in that Performing target detection and target tracking on the image data in sequence to obtain image detection results includes: Performing target detection on the image data to obtain traffic participants in the image data, as well as categories, frame pixel coordinates and sizes corresponding to the traffic participants; Determining the position coordinates of the traffic participant in a world coordinate system based on the frame pixel coordinates and the image calibration matrix; Tracking the traffic participant to determine the image identification and speed of the traffic participant; Determine the attribute information corresponding to the traffic participant based on the category, position coordinates, size, image identifier and speed corresponding to the traffic participant; Determining the image detection result based on the attribute information corresponding to the traffic participant; The image detection result is fused with the radar detection result to obtain a fused perception result, including: According to the attribute information corresponding to the traffic participants in the image detection results and the attribute information corresponding to the traffic participants in the radar detection results, the traffic participants in the image detection results are matched one by one with the traffic participants in the radar detection results to obtain the matched traffic participants; For any one of the corresponding traffic participants, the attribute information of the any one of the traffic participants in the image detection result is fused with the attribute information of the any one of the traffic participants in the radar detection result to obtain a fused perception result corresponding to the any one of the traffic participants.

5. The method according to claim 4, characterized in that Based on the prior information, the fused position coordinates and fused sizes of the traffic participants in the fused perception result are respectively corrected to obtain a corrected fused perception result, including: Determine the position coordinates of each endpoint of the road element in the map according to the prior information; Determine the pixel coordinates of each endpoint on the road element in the image data; Based on the position coordinates of each endpoint on the road element in the map and the pixel coordinates in the image data, as well as the border pixel coordinates corresponding to the traffic participant, the fusion position coordinates and fusion size of the traffic participant are corrected respectively to obtain a corrected fusion perception result.

6. The method according to claim 5, characterized in that When the endpoints on the road element include a first endpoint, a second endpoint, a third endpoint, and a fourth endpoint, based on the position coordinates of the endpoints on the road element in the map and the pixel coordinates in the image data, and the frame pixel coordinates corresponding to the traffic participant, the fusion position coordinates and the fusion size of the traffic participant are respectively corrected to obtain a corrected fusion perception result, including: Based on the border pixel coordinates corresponding to the traffic participant, determine the pixel coordinates of the center point of the lower edge of the border, the border pixel width and the border pixel height; Based on the pixel coordinates corresponding to the first endpoint, the second endpoint, the third endpoint, and the fourth endpoint, respectively, and the pixel coordinates of the center point of the lower edge of the frame, respectively calculate the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the frame, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, and the vertical pixel difference between the second endpoint and the fourth endpoint; Determine the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint based on the position coordinates corresponding to the first endpoint, the second endpoint, the third endpoint, and the fourth endpoint respectively; According to the pixel coordinates of the center point of the lower edge of the frame, the pixel width of the frame, the pixel height of the frame, the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the frame, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, the vertical pixel difference between the second endpoint and the fourth endpoint, the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint, the fusion position coordinates and the fusion size are respectively corrected to obtain corrected position coordinates and corrected size; The corrected fusion perception result is determined according to the corrected position coordinates and the corrected size.

7. The method according to claim 6, characterized in that According to the pixel coordinates of the center point of the lower edge of the frame, the pixel width of the frame, the pixel height of the frame, the horizontal pixel difference and the vertical coordinate difference between the first endpoint and the center point of the lower edge of the frame, the horizontal pixel difference between the first endpoint and the third endpoint, the horizontal pixel difference between the first endpoint and the second endpoint, the vertical pixel difference between the second endpoint and the fourth endpoint, the distance between the first endpoint and the third endpoint, the distance between the first endpoint and the second endpoint, and the distance between the second endpoint and the fourth endpoint, the fusion position coordinates and the fusion size are respectively corrected to obtain the corrected position coordinates and the corrected size, including: Calculate a horizontal correction distance according to a horizontal pixel difference between the first endpoint and the center point of the lower edge of the frame, a horizontal pixel difference between the first endpoint and the third endpoint, and a distance between the first endpoint and the third endpoint; Calculate the longitudinal correction distance according to the longitudinal coordinate difference between the first endpoint and the center point of the lower edge of the frame, the transverse pixel difference between the first endpoint and the second endpoint, and the distance between the first endpoint and the second endpoint; Based on the lateral correction distance and the longitudinal correction distance, the fused position coordinates are corrected to obtain the corrected position coordinates; Calculating a correction width according to the border pixel width, the horizontal pixel difference between the first endpoint and the third endpoint, and the distance between the first endpoint and the third endpoint; Calculate the corrected length according to the frame pixel height, the longitudinal pixel difference between the second endpoint and the fourth endpoint, and the distance between the second endpoint and the fourth endpoint; The fusion size is corrected according to the corrected width and the corrected length to obtain the corrected size.

8. A roadside radar-visual fusion perception device combined with a high-precision map, characterized in that: include: An acquisition unit, used to acquire image data captured by a camera in real time; A segmentation unit, used for performing semantic segmentation on the image data to obtain road elements in the image data; A determination unit, used to determine the prior information of the road element in the map; A detection unit, used to perform target detection and target tracking on the image data in sequence to obtain an image detection result; A fusion unit, used to fuse the image detection result with the radar detection result to obtain a fusion perception result; The correction unit is used to correct the fusion position coordinates and fusion size of the traffic participants in the fusion perception result based on the prior information to obtain a corrected fusion perception result.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images