Roadside universal obstacle sensing method and device based on thunder-vision fusion

Abstract

The invention discloses a roadside universal obstacle sensing method and device based on Leiyu fusion, and relates to the technical field of information, and the method comprises the steps: obtaining image data shot by a camera in real time; performing semantic segmentation and target detection on the image data to obtain foreground information and a detection target in the image data; matching the detection target with a convex hull in the foreground information, and determining a known type of obstacle and an unknown type of obstacle in the image data according to a matching result; performing multi-target tracking on the obstacles of the known type and the obstacles of the unknown type to obtain an image detection result; and fusing the image detection result and the radar detection result, and outputting fusion sensing results corresponding to the known type of obstacle and the unknown type of obstacle respectively. The method can sense an unknown type of obstacle.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and device for sensing general roadside obstacles based on radar-vision fusion. Background Art

[0002] Intelligent connected vehicles are closely integrated with transportation, energy, and cities. In order to promote the high-quality and rapid development of intelligent connected vehicles, my country has proposed a development route of vehicle-road-cloud integration, among which roadside perception can provide detailed information about traffic participants, which is an important part of the development of China's intelligent connected vehicles.

[0003] At present, cameras and millimeter-wave radars are usually used to sense obstacles when performing roadside perception. However, since the visual part of the existing solution uses a supervised learning target detection algorithm, the entire system can only sense obstacles of known types, but has no ability to sense obstacles of unknown types. Summary of the invention

[0004] The present invention provides a method and device for sensing general roadside obstacles based on radar-vision fusion, which are mainly capable of sensing obstacles of unknown types.

[0005] According to a first aspect of an embodiment of the present invention, a method for sensing general roadside obstacles based on radar and visual fusion is provided, comprising:

[0006] Get the image data captured by the camera in real time;

[0007] Performing semantic segmentation and target detection on the image data respectively to obtain foreground information and detection targets in the image data;

[0008] Matching the detected target with the convex hull in the foreground information, and determining known type obstacles and unknown type obstacles in the image data according to the matching result;

[0009] Performing multi-target tracking on the known type of obstacles and the unknown type of obstacles to obtain image detection results;

[0010] The image detection result is fused with the radar detection result, and the fused perception results corresponding to the known type of obstacle and the unknown type of obstacle are output respectively.

[0011] According to a second aspect of an embodiment of the present invention, a general roadside obstacle perception device based on radar and visual fusion is provided, comprising:

[0012] An acquisition unit, used to acquire image data captured by a camera in real time;

[0013] A detection unit, used to perform semantic segmentation and target detection on the image data to obtain foreground information and detection targets in the image data;

[0014] A matching unit, configured to match the detected target with the convex hull in the foreground information, and determine known obstacles and unknown obstacles in the image data according to the matching result;

[0015] A tracking unit, used for performing multi-target tracking on the known type obstacles and the unknown type obstacles to obtain image detection results;

[0016] The fusion unit is used to fuse the image detection result with the radar detection result, and output the fusion perception results corresponding to the known type of obstacle and the unknown type of obstacle respectively.

[0017] According to a third aspect of an embodiment of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0018] Get the image data captured by the camera in real time;

[0019] Performing semantic segmentation and target detection on the image data respectively to obtain foreground information and detection targets in the image data;

[0020] Matching the detected target with the convex hull in the foreground information, and determining known type obstacles and unknown type obstacles in the image data according to the matching result;

[0021] Performing multi-target tracking on the known type of obstacles and the unknown type of obstacles to obtain image detection results;

[0022] The image detection result is fused with the radar detection result, and the fused perception results corresponding to the known type of obstacle and the unknown type of obstacle are output respectively.

[0023] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0024] Get the image data captured by the camera in real time;

[0025] Performing semantic segmentation and target detection on the image data respectively to obtain foreground information and detection targets in the image data;

[0026] Matching the detected target with the convex hull in the foreground information, and determining known type obstacles and unknown type obstacles in the image data according to the matching result;

[0027] Performing multi-target tracking on the known type of obstacles and the unknown type of obstacles to obtain image detection results;

[0028] The image detection result is fused with the radar detection result, and the fused perception results corresponding to the known type of obstacle and the unknown type of obstacle are output respectively.

[0029] The present invention provides a method and device for sensing general obstacles on the roadside based on radar-visual fusion. Compared with the prior art, the method and device can obtain image data captured by a camera in real time, and perform semantic segmentation and target detection on the image data to obtain foreground information and detection targets in the image data. At the same time, the detection targets are matched with the convex hull in the foreground information, and according to the matching results, the known type obstacles and unknown type obstacles in the image data are determined, and then the known type obstacles and the unknown type obstacles are multi-target tracked to obtain image detection results, and finally the image detection results are fused with the radar detection results to output the fusion perception results corresponding to the known type obstacles and the unknown type obstacles. It can be seen that the present invention can identify all known type obstacles and unknown type obstacles in the image data by performing semantic segmentation and target detection on the image data, obtaining foreground information and detection targets in the image data, and matching the detection targets with the convex hull in the foreground information, so as to output the fusion perception results of known type obstacles and unknown type obstacles, thereby realizing the perception of unknown type obstacles and improving the safety of intelligent network-connected driving vehicles.

[0030] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1A schematic diagram of a general roadside obstacle perception method based on radar-visual fusion provided by an embodiment of the present invention is shown;

[0033] Figure 2 The overall process diagram provided by the embodiment of the present invention is shown;

[0034] Figure 3 A schematic diagram of a post-fusion process provided by an embodiment of the present invention is shown;

[0035] Figure 4 A schematic diagram of the structure of a universal roadside obstacle sensing device based on radar and visual fusion provided by an embodiment of the present invention is shown;

[0036] Figure 5 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0039] Since the visual part of the existing solution adopts a supervised learning target detection algorithm, the entire system can only perceive obstacles of known types, but has no ability to perceive obstacles of unknown types.

[0040] In order to overcome the above defects, the embodiment of the present invention provides a roadside general obstacle perception method based on radar and visual fusion, such as Figure 1 As shown, the method includes:

[0041] Step 101: Acquire image data captured by a camera in real time.

[0042] The image data is an image of a road area taken by a camera.

[0043] The embodiment of the present invention is mainly applicable to the scenario of sensing unknown type obstacles and known type obstacles. The execution subject of the embodiment of the present invention is a device or equipment that can sense unknown type obstacles and known type obstacles.

[0044] The overall process of obstacle perception in the embodiment of the present invention is as follows: Figure 2 As shown in the figure, the entire perception process uses cameras and millimeter-wave radars. During the specific perception, the camera will be used to capture the image data of the road in real time, and then the acquired image data will be decoded and preprocessed. The preprocessing operations specifically include denoising, scaling and other operations.

[0045] Step 102: perform semantic segmentation and target detection on the image data to obtain foreground information and detection targets in the image data.

[0046] The detection target is an obstacle of known type, and the foreground information includes obstacles of known type and obstacles of unknown type.

[0047] For the embodiment of the present invention, after acquiring the image data, it is necessary to identify the detection target and foreground information in the image data. For this process, step 102 specifically includes: performing semantic segmentation on the image data to obtain a semantic segmentation result corresponding to the image data; differentiating the semantic segmentation result with the pre-stored static background semantics of the road to obtain the foreground information in the image data; performing target detection on the image data to obtain the detection target in the image data.

[0048] Among them, the road static background semantics specifically includes roads and lane lines.

[0049] Specifically, by performing semantic segmentation on the road image, the static background semantics of the road can be obtained and stored. During real-time perception, the semantic segmentation result is differentiated from the pre-stored static background semantics of the road to obtain the foreground information in the image data. At the same time, the image data is input into a preset target detection model for target detection, and the category, border and pixel coordinates of the obstacles in the image data are output. The pixel coordinates specifically include the pixel coordinates of the upper left corner and the lower right corner of the border.

[0050] It should be noted that the preset target detection model used in the embodiment of the present invention may be a YOLOv5 model or other target detection models, and the embodiment of the present invention does not specifically limit this.

[0051] Step 103: Match the detected target with the convex hull in the foreground information, and determine known type obstacles and unknown type obstacles in the image data according to the matching result.

[0052] The foreground information includes the convex hull and the pixel coordinates of the vertices constituting the convex hull, and the convex hull can be a known type of obstacle or an unknown type of obstacle. In addition, the number of detection targets and convex hulls can be multiple.

[0053] For the embodiment of the present invention, since a supervised learning detection algorithm is used for the detection target, the type of the detection target is known, that is, the detection target is a known type of obstacle. By matching the detection target with the convex hull in the foreground information, all known types of obstacles and unknown types of obstacles in the foreground information can be determined according to the matching result. For this process, step 103 specifically includes: calculating the center position distance between the detection target and the convex hull according to the pixel coordinates of the border corresponding to the detection target and the pixel coordinates of the convex hull; if the center position distance is less than the preset distance, it is determined that the detection target matches the convex hull; if the center position distance is greater than or equal to the preset distance, it is determined that the detection target does not match the convex hull.

[0054] The preset distance may be set according to actual business requirements, and the embodiment of the present invention does not specifically limit this.

[0055] Specifically, according to the pixel coordinates of the vertices on the border corresponding to the detection target and the pixel coordinates of the vertices on the convex hull, the center position of the detection target and the centroid position of the convex hull are determined respectively, and then the center position distance between each detection target and each convex hull is calculated respectively according to the center position of the detection target and the centroid position of the convex hull. For any detection target and any convex hull, if the center position distance between the detection target and the convex hull is less than the preset distance, it means that the detection target matches the convex hull, and the detection target and the convex hull are likely to be the same obstacle. At this time, the convex hull can be determined to be a known type obstacle of the same type as the detection target; on the contrary, if the center position distance between the detection target and the convex hull is greater than or equal to the preset distance, it means that the convex hull does not match the detection target. When it is determined that the convex hull does not match all detection targets, it means that the convex hull is an unknown type obstacle. In this way, all known type obstacles and unknown type obstacles in the image data can be identified in the above manner.

[0056] Furthermore, in order to improve the matching accuracy between the detection target and the convex hull, when the center position distance between the detection target and the convex hull is less than a preset distance, the overlap between the detection target and the convex hull can be further calculated, and based on the overlap, it is finally determined whether the detection target and the convex hull match. For this process, the method includes: if the center position distance is less than the preset distance, the overlap between the detection target and the convex hull is calculated based on the pixel coordinates of the frame corresponding to the detection target and the pixel coordinates of the convex hull; if the overlap is greater than the preset overlap, it is determined that the detection target matches the convex hull; if the overlap is less than or equal to the preset overlap, it is determined that the detection target does not match the convex hull.

[0057] The preset overlap degree may be set according to actual business requirements, and the embodiment of the present invention does not specifically limit this.

[0058] Step 104: Perform multi-target tracking on the known type obstacles and the unknown type obstacles to obtain image detection results.

[0059] In an embodiment of the present invention, after identifying all known type obstacles and unknown type obstacles in the image data, target tracking can be performed on all obstacles to obtain image identification and speed information of the obstacles. For this process, step 104 specifically includes: performing multi-target tracking on the known type obstacles and the unknown type obstacles to obtain the image identification and speed corresponding to the known type obstacles and the unknown type obstacles respectively; performing coordinate conversion on the pixel coordinates of the frame corresponding to the known type obstacles and the pixel coordinates of the convex hull corresponding to the unknown type obstacles respectively to obtain the position coordinates of the known type obstacles and the unknown type obstacles in the world coordinate system respectively; determining the attribute information corresponding to the known type obstacles and the unknown type obstacles respectively according to the image identification, speed, position coordinates and category corresponding to the known type obstacles and the unknown type obstacles respectively; determining the image detection result according to the attribute information corresponding to the known type obstacles and the unknown type obstacles respectively. Among them, the image identification can be specifically an image ID.

[0060] Specifically, by performing multi-target tracking on all known and unknown obstacles, the image identification and speed of each obstacle can be determined. At the same time, in the aforementioned target detection process, the pixel coordinates of the bounding box corresponding to the known obstacle can be obtained, and the foreground information includes the pixel coordinates of the convex hull corresponding to the unknown obstacle. Through the image calibration matrix, the pixel coordinates of the known and unknown obstacles in the image coordinate system are converted into position coordinates in the world coordinate system. In addition, in the aforementioned target detection process, the category of the known obstacle can be determined. Thus, the attribute information of all known and unknown obstacles can be determined, and then the image detection result can be determined.

[0061] It should be noted that when tracking a target, if there are any missing frames, they can be supplemented by prediction.

[0062] Step 105: Fuse the image detection result with the radar detection result, and output fused perception results corresponding to the known type obstacle and the unknown type obstacle respectively.

[0063] In the embodiment of the present invention, after obtaining the image detection results and radar detection results of known type obstacles and unknown type obstacles, it is necessary to fuse the image detection results and radar detection results. For this process, Figure 3 As shown, the method includes: taking the known type obstacle and the unknown type obstacle as respective detection objects, and making one-to-one correspondence between the respective detection objects in the image detection result and the respective detection objects in the radar detection result according to the attribute information corresponding to the respective detection objects in the image detection result and the attribute information corresponding to the respective detection objects in the radar detection result, so as to obtain the respective detection objects after correspondence; for any one of the respective detection objects after correspondence, fusing the attribute information of the any one detection object in the image detection result with the attribute information of the any one detection object in the radar detection result, so as to obtain the fused perception result corresponding to the any one detection object.

[0064] Furthermore, when making one-to-one correspondence between each detection object in the image detection result and each detection object in the radar detection result, the method includes: calculating the similarity between each detection object in the image detection result and each detection object in the radar detection result based on the attribute information corresponding to each detection object in the image detection result and the attribute information corresponding to each detection object in the radar detection result; and making one-to-one correspondence between each detection object in the image detection result and each detection object in the radar detection result based on the similarity to obtain the corresponding detection objects.

[0065] Specifically, before fusion, the radar and camera need to be time synchronized. When the image detection results and radar detection results come in, the two most recent frames of data are selected and the two frames of data are associated and matched, that is, similarity calculation is performed. If the similarity between a detection object in the image detection result and a detection object in the radar detection result is greater than a preset threshold, it means that the two detection objects are the same detection object; on the contrary, if the similarity between a detection object in the image detection result and a detection object in the radar detection result is less than or equal to the preset threshold, it means that the two detection objects are not the same detection object.

[0066] Furthermore, in order to reduce the amount of calculation, the embodiment of the present invention can determine the traveling direction of the detection object according to the speed of the detection object in the radar detection result, and determine the distance threshold according to the traveling direction. If the position distance between a detection object in the image detection result and a detection object in the radar detection result is greater than the distance threshold, it means that the two detection objects are definitely not the same detection object, and no similarity calculation is required; on the contrary, if the position distance between a detection object in the image detection result and a detection object in the radar detection result is less than or equal to the distance threshold, it means that the two detection objects are likely to be the same detection object, and further determination is required through similarity calculation.

[0067] Furthermore, when fusing the attribute information of any detection object in the image detection result with the attribute information in the radar detection result, the method also includes: when the attribute information is an image identifier, determining the fusion identifier corresponding to the any detection object according to the image identifier of the any detection object in the image detection result and the radar identifier in the radar detection result; when the attribute information is speed, weighting the speed of the any detection object in the image detection result and the speed in the radar detection result to determine the fusion speed corresponding to the any detection object; when the attribute information is position coordinates, determining the fusion position coordinates corresponding to the any detection object according to the position coordinates of the any detection object in the image detection result or the position coordinates in the radar detection result; when the attribute information is a category, determining the fusion category corresponding to the any detection object according to the category of the any detection object in the image detection result; and determining the fusion perception result corresponding to the any detection object according to the fusion identifier, the fusion speed, the fusion position coordinates and the fusion category.

[0068] Specifically, when performing frame fusion, the final fusion identification includes image identification and radar identification; the final fusion category trusts the detection result of the image; the final fusion speed trusts the radar more, so the weight of the radar will be higher; the final fusion distance can be determined according to the distance between the sensor and the detection object, the image detection result is more trusted for the near one, and the radar detection result is more trusted for the far one. The longitudinal distance will trust the radar detection result more, and the lateral distance will trust the image detection result more.

[0069] It should be noted that the attribute information in the embodiment of the present invention includes not only image identification, radar identification, speed, category, and position coordinates, but also heading angle, size information, and the like.

[0070] Furthermore, after frame fusion, historical information tracking is performed, and the current fusion data is associated and matched with the previous frame data. Specifically, the association matching can be performed by calculating the similarity, and each detection object is matched one by one, so that the driving trajectory of each detection object can be determined. Furthermore, the detection object on the track is updated, and the trajectory points of the detection object are added. At the same time, the historical detection objects that are not matched are updated, that is, they are predicted or deleted.

[0071] An embodiment of the present invention provides a general roadside obstacle perception method based on radar-vision fusion, which performs semantic segmentation and target detection on image data, obtains foreground information and detection targets in the image data, and matches the detection targets with the convex hull in the foreground information. It can identify all known types of obstacles and unknown types of obstacles in the image data, thereby outputting a fusion perception result of known types of obstacles and unknown types of obstacles, and further realizing the perception of unknown types of obstacles, thereby improving the safety of intelligent connected driving vehicles.

[0072] Further, as Figure 1 In the specific implementation of the present invention, an embodiment of the present invention provides a universal roadside obstacle perception device based on radar and vision fusion, such as Figure 4 As shown, the device includes: an acquisition unit 31, a detection unit 32, a matching unit 33, a tracking unit 34 and a fusion unit 35.

[0073] The acquisition unit 31 may be used to acquire image data captured by a camera in real time.

[0074] The detection unit 32 may be used to perform semantic segmentation and target detection on the image data to obtain foreground information and detection targets in the image data.

[0075] The matching unit 33 may be used to match the detected target with the convex hull in the foreground information, and determine known type obstacles and unknown type obstacles in the image data according to the matching result.

[0076] The tracking unit 34 may be used to perform multi-target tracking on the known type obstacles and the unknown type obstacles to obtain image detection results.

[0077] The fusion unit 35 may be configured to fuse the image detection result with the radar detection result, and output fusion perception results corresponding to the known type obstacle and the unknown type obstacle respectively.

[0078] In a specific application scenario, the detection unit 32 can be specifically used to perform semantic segmentation on the image data to obtain a semantic segmentation result corresponding to the image data; differentiate the semantic segmentation result from the pre-stored static background semantics of the road to obtain foreground information in the image data; perform target detection on the image data to obtain a detection target in the image data.

[0079] In a specific application scenario, the matching unit 33 includes a calculation module and a determination module.

[0080] The calculation module may be used to calculate the center position distance between the detection target and the convex hull according to the pixel coordinates of the frame corresponding to the detection target and the pixel coordinates of the convex hull.

[0081] The determination module may be configured to determine that the detection target matches the convex hull if the center position distance is less than a preset distance.

[0082] The determination module may also be configured to determine that the detection target does not match the convex hull if the center position distance is greater than or equal to the preset distance.

[0083] Furthermore, the calculation module can also be used to calculate the overlap between the detection target and the convex hull according to the pixel coordinates of the frame corresponding to the detection target and the pixel coordinates of the convex hull if the center position distance is less than a preset distance.

[0084] The determination module may also be configured to determine that the detection target matches the convex hull if the overlap degree is greater than a preset overlap degree.

[0085] The determination module may also be configured to determine that the detection target does not match the convex hull if the overlap degree is less than or equal to the preset overlap degree.

[0086] Furthermore, the determination module may also be configured to determine that the convex hull is a known type of obstacle if the convex hull matches the detection target; and determine that the convex hull is an unknown type of obstacle if the convex hull does not match the detection target.

[0087] In a specific application scenario, the tracking unit 34 can be specifically used to perform multi-target tracking on the known type obstacle and the unknown type obstacle to obtain the image identification and speed corresponding to the known type obstacle and the unknown type obstacle respectively; perform coordinate conversion on the pixel coordinates of the border corresponding to the known type obstacle and the pixel coordinates of the convex hull corresponding to the unknown type obstacle respectively to obtain the position coordinates of the known type obstacle and the unknown type obstacle in the world coordinate system respectively; determine the attribute information corresponding to the known type obstacle and the unknown type obstacle respectively according to the image identification, speed, position coordinates and category corresponding to the known type obstacle and the unknown type obstacle respectively; determine the image detection result according to the attribute information corresponding to the known type obstacle and the unknown type obstacle respectively.

[0088] In a specific application scenario, the fusion unit 35 includes: a corresponding module and a fusion module.

[0089] The corresponding module can be used to take the known type obstacles and the unknown type obstacles as respective detection objects, and according to the attribute information corresponding to each detection object in the image detection result and the attribute information corresponding to each detection object in the radar detection result, make one-to-one correspondence between each detection object in the image detection result and each detection object in the radar detection result to obtain the corresponding detection objects.

[0090] The fusion module can be used to fuse the attribute information of any one of the corresponding detection objects in the image detection result with its attribute information in the radar detection result to obtain a fused perception result corresponding to the any one detection object.

[0091] Furthermore, the corresponding module can be specifically used to calculate the similarity between each detection object in the image detection result and each detection object in the radar detection result based on the attribute information corresponding to each detection object in the image detection result and the attribute information corresponding to each detection object in the radar detection result; based on the similarity, each detection object in the image detection result is matched one-to-one with each detection object in the radar detection result to obtain the corresponding detection objects.

[0092] Furthermore, the fusion module can be specifically used to determine the fusion identifier corresponding to any one of the detection objects according to the image identifier of the any one of the detection objects in the image detection result and the radar identifier of the any one of the detection objects in the radar detection result when the attribute information is an image identifier; when the attribute information is a speed, weight the speed of the any one of the detection objects in the image detection result and the speed of the any one of the detection objects in the radar detection result to determine the fusion speed corresponding to the any one of the detection objects; when the attribute information is a position coordinate, determine the fusion position coordinate corresponding to the any one of the detection objects according to the position coordinates of the any one of the detection objects in the image detection result or the position coordinates of the radar detection result; when the attribute information is a category, determine the fusion category corresponding to the any one of the detection objects according to the category of the any one of the detection objects in the image detection result; determine the fusion perception result corresponding to the any one of the detection objects according to the fusion identifier, the fusion speed, the fusion position coordinate and the fusion category.

[0093] It should be noted that for other corresponding descriptions of the functional modules involved in the universal roadside obstacle perception device based on radar and visual fusion provided in the embodiment of the present invention, please refer to Figure 1 The corresponding description of the method shown will not be repeated here.

[0094] Based on the above Figure 1 The method shown, accordingly, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: acquiring image data captured by a camera in real time; performing semantic segmentation and target detection on the image data respectively, to obtain foreground information and detection targets in the image data; matching the detection targets with the convex hull in the foreground information, and determining known type obstacles and unknown type obstacles in the image data according to the matching results; performing multi-target tracking on the known type obstacles and the unknown type obstacles to obtain image detection results; fusing the image detection results with the radar detection results, and outputting the fused perception results corresponding to the known type obstacles and the unknown type obstacles respectively.

[0095] Based on the above Figure 1 The method shown and Figure 4 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of an electronic device, such as Figure 5As shown, the electronic device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 implements the following steps when executing the program: acquiring image data captured by a camera in real time; performing semantic segmentation and target detection on the image data respectively to obtain foreground information and detection targets in the image data; matching the detection targets with the convex hull in the foreground information, and determining known type obstacles and unknown type obstacles in the image data according to the matching results; performing multi-target tracking on the known type obstacles and the unknown type obstacles to obtain image detection results; fusing the image detection results with the radar detection results, and outputting fused perception results corresponding to the known type obstacles and the unknown type obstacles respectively.

[0096] The embodiments of the present invention perform semantic segmentation and target detection on image data, obtain foreground information and detection targets in the image data, and match the detection targets with the convex hull in the foreground information. This can identify all known types of obstacles and unknown types of obstacles in the image data, thereby outputting a fusion perception result of known types of obstacles and unknown types of obstacles, thereby realizing the perception of unknown types of obstacles and improving the safety of intelligent connected driving vehicles.

[0097] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0098] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general roadside obstacle perception method based on radar and visual fusion, characterized in that: include: Get the image data captured by the camera in real time; Performing semantic segmentation and target detection on the image data respectively to obtain foreground information and detection targets in the image data; Matching the detected target with the convex hull in the foreground information, and determining known type obstacles and unknown type obstacles in the image data according to the matching result; Performing multi-target tracking on the known type of obstacles and the unknown type of obstacles to obtain image detection results; The image detection result is fused with the radar detection result, and the fused perception results corresponding to the known type of obstacle and the unknown type of obstacle are output respectively.

2. The method according to claim 1, characterized in that The performing semantic segmentation and target detection on the image data respectively to obtain foreground information and detection targets in the image data includes: Performing semantic segmentation on the image data to obtain a semantic segmentation result corresponding to the image data; Differentiating the semantic segmentation result from the pre-stored static background semantics of the road to obtain foreground information in the image data; Performing target detection on the image data to obtain a detected target in the image data; Matching the detected target with the convex hull in the foreground information includes: Calculating the center position distance between the detection target and the convex hull according to the pixel coordinates of the frame corresponding to the detection target and the pixel coordinates of the convex hull; If the center position distance is less than a preset distance, determining that the detection target matches the convex hull; If the center position distance is greater than or equal to the preset distance, it is determined that the detection target does not match the convex hull.

3. The method according to claim 2, characterized in that The method further comprises: If the center position distance is less than the preset distance, the overlap between the detection target and the convex hull is calculated according to the pixel coordinates of the frame corresponding to the detection target and the pixel coordinates of the convex hull; If the overlap is greater than a preset overlap, determining that the detection target matches the convex hull; If the overlap degree is less than or equal to the preset overlap degree, it is determined that the detection target does not match the convex hull.

4. The method according to claim 1, characterized in that Determining known obstacles and unknown obstacles in the image data according to the matching result includes: If the convex hull matches the detection target, determining that the convex hull is an obstacle of a known type; If the convex hull does not match the detection target, it is determined that the convex hull is an obstacle of unknown type.

5. The method according to claim 1, characterized in that Performing multi-target tracking on the known type of obstacles and the unknown type of obstacles to obtain image detection results includes: Performing multi-target tracking on the known type of obstacles and the unknown type of obstacles to obtain image identifiers and speeds corresponding to the known type of obstacles and the unknown type of obstacles respectively; Performing coordinate conversion on the pixel coordinates of the bounding box corresponding to the known type obstacle and the pixel coordinates of the convex hull corresponding to the unknown type obstacle, respectively, to obtain the position coordinates of the known type obstacle and the unknown type obstacle in the world coordinate system; Determine attribute information corresponding to the known type of obstacle and the unknown type of obstacle respectively according to the image identifier, speed, position coordinate and category corresponding to the known type of obstacle and the unknown type of obstacle respectively; The image detection result is determined according to the attribute information corresponding to the known type of obstacle and the unknown type of obstacle respectively.

6. The method according to claim 1, characterized in that The image detection result is integrated with the radar detection result, and the integrated perception results corresponding to the known type of obstacle and the unknown type of obstacle are outputted, including: The known type obstacle and the unknown type obstacle are taken as detection objects, and according to the attribute information corresponding to each detection object in the image detection result and the attribute information corresponding to each detection object in the radar detection result, each detection object in the image detection result is matched with each detection object in the radar detection result one by one to obtain each detection object after matching; For any one of the corresponding detection objects, the attribute information of the any one detection object in the image detection result is fused with the attribute information of the any one detection object in the radar detection result to obtain a fused perception result corresponding to the any one detection object.

7. The method according to claim 6, characterized in that According to the attribute information corresponding to each detection object in the image detection result and the attribute information corresponding to each detection object in the radar detection result, each detection object in the image detection result is matched with each detection object in the radar detection result one by one to obtain each detection object after matching, including: Calculate the similarity between each detection object in the image detection result and each detection object in the radar detection result according to the attribute information corresponding to each detection object in the image detection result and the attribute information corresponding to each detection object in the radar detection result; According to the similarity, each detection object in the image detection result is matched with each detection object in the radar detection result one by one to obtain each detection object after matching; Fusing the attribute information of any one of the detection objects in the image detection result with the attribute information of any one of the detection objects in the radar detection result to obtain a fused perception result corresponding to the any one of the detection objects, including: When the attribute information is an image identifier, determining a fusion identifier corresponding to the any one detection object according to the image identifier of the any one detection object in the image detection result and the radar identifier of the any one detection object in the radar detection result; When the attribute information is speed, weighting the speed of the arbitrary detection object in the image detection result and the speed of the arbitrary detection object in the radar detection result to determine the fusion speed corresponding to the arbitrary detection object; When the attribute information is position coordinates, determining the fused position coordinates corresponding to the any one detection object according to the position coordinates of the any one detection object in the image detection result or the position coordinates of the any one detection object in the radar detection result; When the attribute information is a category, determining a fusion category corresponding to the any one detection object according to the category of the any one detection object in the image detection result; A fusion perception result corresponding to any one of the detection objects is determined according to the fusion identifier, the fusion speed, the fusion position coordinates and the fusion category.

8. A universal roadside obstacle perception device based on radar and visual fusion, characterized in that: include: An acquisition unit, used to acquire image data captured by a camera in real time; A detection unit, used to perform semantic segmentation and target detection on the image data to obtain foreground information and detection targets in the image data; A matching unit, used to match the detected target with the convex hull in the foreground information, and determine known type obstacles and unknown type obstacles in the image data according to the matching result; A tracking unit, used for performing multi-target tracking on the known type obstacles and the unknown type obstacles to obtain image detection results; The fusion unit is used to fuse the image detection result with the radar detection result, and output the fusion perception results corresponding to the known type of obstacle and the unknown type of obstacle respectively.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images