A multi-sensor-based roadside perception method, apparatus, device and medium

Abstract

Embodiments of the present application provide a roadside sensing method and device based on multiple sensors, equipment and medium, the method comprising: obtaining the to-be-fused sensing data of a sensing area; determining the data association result of the to-be-fused sensing data and a pre-created fusion target list according to the to-be-fused sensing data and the fusion target list; updating the fusion target list according to the data association result; and determining the fusion target information of the sensing area according to the associated sensing data corresponding to each fusion target in the fusion target list. By using the method, only visual sensing data is captured in the target creation area, visual sensing data and radar sensing data are captured in the data fusion area, and only radar sensing data is captured in other areas, which is equivalent to performing data fusion only in the data fusion area where the camera and radar detection accuracy are both high, thereby improving the target fusion effect. Moreover, the farthest detection distance is not limited to the detection distance of the camera, thereby increasing the range of target fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of roadside perception, and in particular to a roadside perception method based on multiple sensors, a roadside perception device, equipment and a medium. BACKGROUND

[0002] Roadside perception is to use cameras, millimeter wave radars, lidar sensors, and combine roadside edge computing, and the ultimate goal is to realize instantaneous intelligent perception of traffic participants and road conditions on the road section. Through the fusion of cameras and millimeter wave radars, target detection, positioning and tracking are realized. Common millimeter wave radar and camera fusion is mainly based on cameras, and the target and target attributes detected by the radar are associated and fused on the entire image.

[0003] However, in the prior art, the perception data captured by the camera and the radar detection perception data are associated in the entire range. Since the radar is prone to multiple clustering points for large volume targets, for example, a large truck is detected as multiple targets, causing radar false detection, missed detection, etc., which causes radar and camera association abnormalities, resulting in poor fusion effect (e.g., the position, speed, heading angle, etc. of the fused target is updated incorrectly). In addition, due to the limited detection distance of the camera, the maximum detection distance of the entire detection range is limited by the camera. SUMMARY

[0004] The embodiments of the present application provide a roadside perception method based on multiple sensors, a device, equipment and a medium, which realizes improved target fusion effect and increased target fusion range.

[0005] In a first aspect, the embodiments of the present application provide a roadside perception method based on multiple sensors, comprising:

[0006] Obtaining to-be-fused perception data of a perception area;

[0007] According to the to-be-fused perception data and a pre-created fusion target list, determining a data association result of the to-be-fused perception data and the fusion target list;

[0008] According to the data association result, updating the fusion target list;

[0009] According to the associated perception data corresponding to each fusion target in the fusion target list, determining the fusion target information of the perception area;

[0010] The perception area is divided into a target creation area, a data fusion area and other areas in sequence according to the target moving direction, the target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other area corresponds to radar perception data.

[0011] In a second aspect, an embodiment of the present application provides a multi-sensor based roadside perception device, comprising:

[0012] a data acquisition module configured to acquire to-be-fused perception data of a perception area;

[0013] a target association module configured to determine a data association result of the to-be-fused perception data and a fusion target list according to the to-be-fused perception data and the pre-created fusion target list;

[0014] a data fusion module configured to update the fusion target list according to the data association result;

[0015] an information determination module configured to determine fusion target information of the perception area according to associated perception data corresponding to each fusion target in the fusion target list;

[0016] The perception area is sequentially divided into a target creation area, a data fusion area and other areas according to a target moving direction, the target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other areas correspond to radar perception data.

[0017] In a third aspect, an embodiment of the present application further provides an edge computing device, comprising:

[0018] at least one processor; and

[0019] a memory connected with the at least one processor in communication; wherein

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-sensor based roadside perception method according to the first aspect.

[0021] In a fourth aspect, an embodiment of the present application further provides a storage medium containing computer executable instructions, which are used to execute the multi-sensor based roadside perception method according to the first aspect when executed by a computer processor.

[0022] The embodiment of the present application provides a roadside sensing method, device and equipment based on multiple sensors and a medium, which comprises the following steps: firstly, obtaining to-be-fused sensing data of a sensing area; secondly, determining a data association result of the to-be-fused sensing data and a pre-created fusion target list according to the to-be-fused sensing data and the fusion target list; then, updating the fusion target list according to the data association result; finally, determining fusion target information of the sensing area according to associated sensing data corresponding to each fusion target in the fusion target list; the sensing area is divided into a target creation area, a data fusion area and other areas in turn according to a target moving direction, the target creation area corresponds to visual sensing data, the data fusion area corresponds to visual sensing data and radar sensing data, and the other areas correspond to radar sensing data. According to the technical solution, the sensing area is divided into three areas according to the detection accuracy of different sensors, visual sensing data is captured in the target creation area, visual sensing data and radar sensing data are captured in the data fusion area, and radar sensing data is captured in the other areas; in other words, the fusion of visual sensing data and radar sensing data is started in the data fusion area where the detection accuracy of the camera and the radar is high, so that the association exception of the radar and the camera is avoided, the accuracy of the target fusion information is ensured, and the target fusion effect is improved. In addition, the radar sensing area is set in the other areas, and the farthest detection distance of the sensing area is no longer limited by the detection distance of the camera due to the long detection distance of the radar, so that the range of the target fusion is increased, and the target fusion effect is further improved.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of a roadside sensing method based on multiple sensors provided by the first embodiment of the present application;

[0026] Figure 2 An example diagram of a sensing area in the execution of a roadside sensing method based on multiple sensors provided by the first embodiment of the present application;

[0027] Figure 3A flowchart of another multi-sensor-based roadside perception method provided for Embodiment Two of the present application is shown in FIG. 2.

[0028] Figure 4 A structural diagram of a multi-sensor-based roadside perception device provided for Embodiment Three of the present application is shown in FIG. 3.

[0029] Figure 5 A structural diagram of an edge computing device provided for Embodiment Four of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0030] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should belong to the scope of protection of the present application.

[0031] It should be noted that the terms "original", "target" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment One

[0033] Figure 1 A flowchart of a multi-sensor-based roadside perception method provided for Embodiment One of the present application is shown in FIG. 1. The method can be applicable to the case of camera and millimeter wave radar-based roadside perception fusion. The method can be performed by a multi-sensor-based roadside perception device, which can be realized in the form of hardware and / or software, and can be configured in an edge computing device.

[0034] In the prior art, the camera-captured visual perception data and the radar-captured radar perception data are associated in the entire perception area. Due to the fact that the radar is prone to generate multiple clustering points for a target with a large volume, for example, three clustering points are generated for a large truck, and the truck is detected as three targets, two of which are false targets, causing radar false detection, missed detection, and the like. When the two false targets are mapped to the image captured by the camera, the radar and the camera are abnormally associated, resulting in poor fusion effect, for example, the position, speed, and heading angle of the fused target are incorrectly updated.

[0035] To avoid the above problems, in the embodiment, the perception area is divided into three areas, which are sequentially divided into a target creation area, a data fusion area, and other areas according to the moving direction of the target. The target creation area only obtains visual perception data from the camera, the data fusion area obtains radar perception data from the radar, and the other area only obtains radar perception data from the radar. This setting fully considers the detection accuracy characteristics of the camera and the radar. The camera can well identify the target and the static attribute information of the target, such as the vehicle model, color, license plate, and the like, in a relatively close area. The radar has a blind area in a relatively close area, and the radar has higher accuracy than the camera in a relatively far range. The radar can well identify the target and the dynamic attribute information of the target, such as the speed, position, and heading angle of the vehicle, in an area outside the blind area. It can be understood that for the target creation area in a relatively close area, the camera has higher detection accuracy and the radar is in a blind area, so only the visual perception data captured by the camera is obtained. For the data fusion area with high detection accuracy of the camera and the radar, the visual perception data captured by the camera and the radar perception data captured by the radar are obtained. For the other area in a relatively far range with high detection accuracy of the radar and limited detection accuracy of the camera, the radar perception data captured by the radar is obtained.

[0036] In the embodiment, to implement the multi-sensor-based roadside perception method, the required hardware devices include three types of devices: millimeter wave radar, camera, and edge computing device. When installing the millimeter wave radar and the camera, the direction is toward the tail of the vehicle. The fused target trajectory has a single-direction feature, that is, the target driving direction is from the target creation area to the data fusion area, and finally runs to the other area. In the embodiment, the data captured by the radar is referred to as radar perception data, and the data captured by the camera is referred to as visual perception data. It should be noted that the embodiment is not limited to the camera, but can also be other visual acquisition sensors, and is not limited to the millimeter wave radar, but can also be a laser radar.

[0037] It can also be understood that some attributes of the target itself, such as vehicle type, vehicle color, license plate, etc. are recognized by the camera when the target creation area is created. These attributes are constant and are carried to the fusion perception area and other areas. The position and speed detected by the radar are more accurate, and the subsequent attributes are maintained by the radar.

[0038] Among them, the perception area is divided into three areas, namely target creation area, data fusion area and other area. The target creation area: in this area, the camera detects the target to create a fusion target, and the reliable type is given to the target which is difficult to detect for pedestrians and non-motor vehicles. Large vehicles will not split false targets. The radar in this area generally cannot detect and will not create a target. Data fusion area: the calibration points in this area are dense and the range is small, which increases the calibration accuracy and simplicity, greatly improves the reliability of association, and the target in the data fusion area is maintained by radar and camera. Other areas: the target running out of the data fusion area will continue to be maintained by the radar with the type, license plate, vehicle color, and specific attributes given by the camera. At this time, the fusion target has reliable speed, position, and heading angle attributes, and the maximum detection distance is determined by the radar, which greatly increases the perception range.

[0039] Among them, the size of the three areas in the perception area can be determined according to the actual situation. For example, if in the city expressway, the speed of the target is relatively low, the creation target area is within 0-30 meters to detect the target, and the data fusion area can be divided small. At this time, the dot is more dense, because the speed is low at this time, the precision difference caused by the delay of the data transmitted by the radar is small, and the data fusion area is small. If in the scene of high speed, the speed of the target on the highway is fast, at this time, the error caused by the delay is relatively large, if the data fusion area is small, the target just comes to the data fusion area, and it has not been processed yet. It has gone out of the data fusion area, so in order to ensure data fusion, the data fusion area needs to be divided larger.

[0040] Exemplary, Figure 2 An example diagram of a perception area in a kind of road side perception method based on multi-sensor for embodiment one of the application is provided. As Figure 2 As shown, in order to identify the target in the perception area 1, radar and camera are installed on the rod, and the perception area is divided into target creation area 11, data fusion area 12 and other area 13. Among them, the target creation area 11 is only captured by the camera to perceive data, the data fusion area 12 is captured by the camera and radar to perceive data, and the other area 13 is captured by the radar to perceive data.

[0041] As Figure 1As shown, the multi-sensor-based roadside perception method provided in this embodiment can specifically include the following steps:

[0042] S110, acquire the to-be-fused perception data of the perception area.

[0043] In this embodiment, the perception area is divided into a target creation area, a data fusion area, and other areas in turn according to the target moving direction, the target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other area corresponds to radar perception data. The target detected in this embodiment is not specifically limited, for example, it can be a vehicle, a pedestrian, etc.

[0044] The principle of radar capturing radar perception data is that the radar first performs electromagnetic wave signal processing, gives the speed, position, and other original point cloud information of the detected target based on Doppler shift, and then outputs radar detection target structured data through clustering and tracking algorithm, which is recorded as radar perception data. It is considered that this step of calculation of the radar can exist in the radar itself.

[0045] The principle of camera capturing visual perception data is that the camera process pulls the camera stream based on GStream (GStreamer is an open source multimedia framework for building streaming applications, used to process multimedia data in various formats), transmits the image data to the target detection algorithm, for example, to the single-stage target detection algorithm YOLOV5, outputs the detection result, and uses the target tracking algorithm, for example, the bytetrack algorithm, to give the tracking implementation, and outputs the camera detection target structured data, which is recorded as visual perception data. This step of calculation can exist in the edge computing device.

[0046] In this embodiment, the to-be-fused perception data output by the radar and the camera are different, the radar perception data captured by the radar is an identification number (ID), a position, and a speed. The visual perception data determined according to the image captured by the camera is target category, vehicle category, vehicle color, license plate, etc. information, and can also detect the position of the target on the image, but will not output how far the target is from the camera, nor will it output the speed. Therefore, the data captured by the radar and the camera need to be combined to jointly maintain the attributes of the detected target.

[0047] The to-be-fused perception data captured by the radar and the camera is given to the edge computing device for data fusion to maintain the attributes of the detected target. It needs to be clear that before data fusion, it can also be judged whether the to-be-fused perception data received by the edge computing device is empty data. If it is empty data, it can be discarded, and new captured to-be-fused perception data can be continuously received. In addition, the to-be-fused data can also be stored to facilitate offline debugging when problems are found later.

[0048] S120, determining the data association result of the to-be-fused perception data and the fusion target list according to the to-be-fused perception data and the pre-created fusion target list.

[0049] In the embodiment, after obtaining the to-be-fused perception data, the attributes of the target are maintained based on the to-be-fused perception data. In the embodiment, the attribute maintenance of the target is represented in the form of a tracking list, denoted as a fusion target list. The fusion target list contains the fusion target and the associated attributes of the fusion target. It can be understood that the fusion target list is empty when it is initially created. With the continuous transmission of the to-be-fused perception data, the fusion target and the associated attributes in the fusion target list are continuously updated.

[0050] The to-be-fused perception data includes visual perception data and radar perception data, and the visual perception data or the radar perception data needs to be maintained in the fusion target list. In the embodiment, different data association strategies are used in different areas for data fusion. Fusion mainly includes association and update, that is, different association strategies and update strategies are implemented in different areas, and the present step mainly relates to how to perform association.

[0051] In the embodiment, it is necessary to determine whether the fusion target list is empty. If the fusion target list is empty, the to-be-fused perception data is unassociated perception data, which is the data association result of the to-be-fused perception data and the fusion target list. If the fusion target list is not empty, the data association result of the to-be-fused perception data and the fusion target list can be determined according to a preset data association strategy. The preset data association strategy includes a first data association strategy, a second data association strategy and a third data association strategy. The first data association strategy refers to association through an identification number. First, ID association is performed, and this step of association depends on the tracking of the original sensor. That is, the fusion target has associated the to-be-fused perception data (radar or camera target) before, and as long as the heading angle difference, position difference and speed difference of the fusion target and the measurement value are within a certain threshold range, the to-be-fused perception data is associated.

[0052] According to the foregoing description, after the first step of association, three kinds of association results are generated, one is a target association pair successfully associated, the other is unassociated fusion target perception data, and the other is unassociated fusion target. These results are recorded as the first association result. If a new perception target comes, the previous step is not successfully associated, that is, the ID association does not work, and at this time, the second step of association is performed. The second step of association refers to Intersection Over Union (IOU) association. For example, the radar perception data is previously transmitted, the radar coordinates xy of the radar perception data are converted into pixel coordinates UV, and the UV information is also maintained in the fusion target list. The coordinate information of the newly transmitted radar perception data is associated with the pixel coordinate information of the maintained fusion target, that is, the IOU association of the fusion region. Each target has a rectangular frame on the image, and the pixel coordinates of the newly transmitted radar perception data are also punched on the frame. After punching, a rectangle is also generated. There is an overlapping region between the two rectangular frames. The area of the overlapping region and the proportion of the area of the two rectangles are taken as the intersection over union. According to the intersection over union, data fusion of the data fusion region is performed.

[0053] According to the foregoing description, other regions consider that the camera detection is not detected, or in other words, there is no visual perception data, so the third data fusion strategy of the third step, that is, distance association, is used. The so-called distance association can be understood as that the perception target and the fusion target have a straight line distance in other regions. It can be simply considered that who is close to whom is considered to be the same target.

[0054] After data association, three kinds of association results are generated, one is a target association pair successfully associated, the other is unassociated fusion target perception data, and the other is unassociated fusion target. These results are recorded as the data association result. For example, it is assumed that 10 radar perception data are transmitted, and there are 10 fusion targets in the fusion target list, of which 4 fusion targets can be associated with 4 radar perception data. The data association result is 6 unassociated perception data, 4 target association pairs, and 6 unassociated fusion targets.

[0055] S130, updating the fusion target list according to the data association result.

[0056] In this embodiment, after association, the association pair is updated. For example, it is assumed that fusion target 1 and perception target B of radar perception data are associated, and after association, the information of perception target B, such as position, speed, and heading angle, is copied to fusion target 1.

[0057] The data association result can include a target association pair, unassociated perception data, and unassociated fusion targets. The target association pair can be understood as that the to-be-perceived fusion data and the fusion target are successfully associated. The unassociated perception data can be understood as some perception data outside the fusion. The unassociated fusion target refers to a fusion target in the fusion target list except for the successfully associated fusion target. The target attribute of the fusion target in the fusion target list can be updated according to the target association pair in the data association result. It can be considered that the target attribute of the to-be-fused perception data corresponding to the associated to-be-fused perception data is updated to the fusion target, and the fusion target is maintained to continue to exist. In this embodiment, after the target association pair is determined, the attribute of the target association pair needs to be updated, that is, the attribute of the fusion target corresponding to the to-be-fused perception data is determined according to the to-be-fused perception data corresponding to the target association pair.

[0058] For the unassociated to-be-fused perception data, a fusion target is created when a certain condition is met, and the created new fusion target is placed in the maintained fusion target list. For the unassociated fusion target, the operation to be performed is to judge whether it has expired, that is, whether the fusion target has really disappeared from the monitored perception area. If the fusion target disappears, it needs to be removed. If the disappearance condition is not met, the fusion target is still in the fusion target list and continues to be maintained. The information of the fusion target is used again in the next association.

[0059] In the fusion, it is first identified whether the visual perception data or the radar perception data is used. If the visual perception data is used, the visual perception data is converted in the coordinate system, and the image UV coordinate is converted into the radar coordinate XY. Because the joint calibration is performed, the pixel coordinate is calibrated with the xy coordinate. After the calibration, there is a mapping relationship between the visual perception data and the radar perception data. After the conversion, a fusion list is maintained, and the visual perception data is fused with the fusion target list. Similarly, if the radar perception data is used, the radar perception data is converted in the coordinate system, which will not be described in detail here.

[0060] S140, determining the fusion target information of the perception area according to the associated perception data corresponding to each fusion target in the fusion target list.

[0061] Because the roadside perception is used for the roadside device V2X, the V2X uses the latitude and longitude information. In this embodiment, after the fusion target list is updated, the associated perception data corresponding to each fusion target in the fusion target list is converted into a set coordinate system according to the required coordinate system, and the corresponding latitude and longitude, id, heading angle, speed, target type, license plate, vehicle color and other information are output as the fusion target information of the perception area.

[0062] It needs to be known that before the fusion target information is determined according to the fusion target list, Kalman filtering can also be performed on the associated perception data corresponding to each fusion target. Considering that the position point of the target detected by the sensor itself may exist jitter, when mapped to a debugging interface or other display interface, it is found that the position is shaking up and down. For example, the real vehicle is straight, and the output point on the image or interface may be a jagged shape, so Kalman filtering is needed to perform trajectory smoothing processing.

[0063] The embodiment of the application provides a roadside perception method based on multiple sensors, which comprises the following steps: firstly, obtaining to-be-fused perception data of a perception area; secondly, determining a data association result of the to-be-fused perception data and a fusion target list according to the to-be-fused perception data and the pre-created fusion target list; thirdly, updating the fusion target list according to the data association result; and finally, determining fusion target information of the perception area according to associated perception data corresponding to each fusion target in the fusion target list. The perception area is divided into a target creation area, a data fusion area and other areas in turn according to a target moving direction, the target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other areas correspond to radar perception data. The technical scheme has the advantages that the perception area is divided into three areas according to the detection accuracy of different sensors, the visual perception data is only captured in the target creation area, the visual perception data and the radar perception data are captured in the data fusion area, and the radar perception data is only captured in the other areas; in other words, the fusion of the visual perception data and the radar perception data is only started in the data fusion area where the detection accuracy of the camera and the radar is high, so that the association abnormality of the radar and the camera is avoided, the accuracy of the target fusion information is ensured, and the target fusion effect is improved. In addition, the radar perception area is set in the other areas, and the farthest detection distance of the perception area is no longer limited by the detection distance of the camera due to the long detection distance of the radar, so that the range of the target fusion is increased, and the target fusion effect is further improved.

[0064] Embodiment two

[0065] Figure 3 The flowchart of another roadside perception method based on multiple sensors provided by the embodiment two of the application is shown, and the embodiment is a further optimization of the above-mentioned embodiment. In the embodiment, the step of "determining the data association result of the to-be-fused perception data and the fusion target list according to the to-be-fused perception data and the pre-created fusion target list" is further limited and optimized, the step of "updating the fusion target list according to the data association result" is further limited and optimized, and the step of "determining the fusion target information of the perception area according to the associated perception data corresponding to each fusion target in the fusion target list" is further limited and optimized.

[0066] As Figure 3 shown in the following embodiments, the present application provides a multi-sensor based roadside perception method, which specifically comprises the following steps:

[0067] S210, obtaining the to-be-fused perception data of the perception area.

[0068] S220, if the fusion target list is empty, taking the to-be-fused perception data as unassociated perception data as the data association result of the to-be-fused perception data and the data of the fusion target list.

[0069] Specifically, if the fusion target list is empty, data fusion cannot be performed, and therefore, the to-be-fused perception data as unassociated perception data is taken as the data association result.

[0070] S230, if the fusion target list is not empty, determining the data association result of the to-be-fused perception data and the fusion target list according to a preset data association strategy.

[0071] The preset data association strategy includes a first data association strategy, a second data association strategy, and a third data association strategy. The first data association strategy refers to association through an identification number. First, ID association is performed, and this step of association depends on the tracking of the original sensor. That is, if the to-be-fused perception data (radar or camera target) has been associated with the fusion target before, as long as the heading angle difference, position difference, and speed difference of the fusion target and the measurement value are within a certain threshold range, the to-be-fused perception data is associated with the fusion target.

[0072] After the above description, three kinds of association results will be generated after the first step of association, one is a successfully associated target association pair, the other is unassociated to-be-fused perception data, and the other is unassociated fusion target. These results are recorded as the first association result. If a new perception target comes in and is not successfully associated in the previous step, that is, ID association does not work, second-step association is performed at this time. The second-step association refers to Intersection Over Union (IOU) association. For example, radar perception data is previously transmitted, the radar coordinates xy of the radar perception data are converted into pixel coordinates UV, and the UV information of the maintained fusion target list also has UV information. The coordinate information of the newly transmitted radar perception data is associated with the pixel coordinate information of the maintained fusion target, that is, the IOU association of the fusion area. Each target has a rectangular frame on the image, and the pixel coordinates of the newly transmitted radar perception data are also punched onto the frame. After punching, a rectangular frame is also generated. There is an overlapping area between the two rectangular frames, and the area of the overlapping area is taken as the intersection over union of the area of the two rectangular frames. Data fusion is performed according to the intersection over union.

[0073] According to the above description, other areas consider that the camera detection is not detected, or that there is no visual perception data, so the third data fusion strategy in the third step is used, that is, distance association. The so-called distance association can be understood as that the perception target and the fusion target have a straight line distance when in other areas, which can be simply considered as who is close to who, and it is considered to be the same target.

[0074] S240, updating the target attribute of the fusion target association in the fusion target list according to the target association pair in the data association result.

[0075] This step can be considered as updating the association pair, and the target attribute corresponding to the to-be-fused perception data associated is updated to the fusion target, and is used to maintain the existence of the fusion target. In this embodiment, after determining the target association pair, the attribute of the target association pair needs to be updated, that is, according to the to-be-fused perception data corresponding to the target association pair, the attribute of the fusion target corresponding to the to-be-fused perception data is determined.

[0076] It can be understood that the to-be-fused perception data corresponding to the target association pair can be visual perception data or radar perception data. If the to-be-fused perception data corresponding to the target association pair is visual perception data, the static attribute in the visual perception data is used as the to-be-added target attribute, and at the same time, some dynamic attributes such as the position, speed, heading angle and pixel coordinate of the target vehicle can be determined according to the data captured by the camera, but these dynamic attributes have large errors and need to be fused with radar perception data.

[0077] If the to-be-fused perception data corresponding to the target association pair is radar perception data, the dynamic attribute in the radar perception data is used as the to-be-added target attribute. At the same time, some dynamic attributes can be determined according to the data captured by the camera, and at this time, the dynamic attribute corresponding to the radar perception data can be fused with the dynamic attribute corresponding to the associated visual perception data.

[0078] At the same time of determining the target attribute to be added, the fusion target corresponding to the target association pair is determined from the fusion target list. The target attribute can be associated with the fusion target to realize the addition operation of the target attribute.

[0079] S250, creating a new fusion target according to the unassociated perception data in the data association result and storing it into the fusion target list.

[0080] In the prior art, the camera-captured visual perception data and the radar-captured radar perception data are associated in the entire perception area. Due to the fact that the radar is prone to generate multiple clustering points for a target with a large volume, for example, a large truck generates three clustering points, and the truck is detected as three targets, two of which are false targets, causing radar false detection, missed detection, and the like. When the two false targets are mapped to the image captured by the camera, the radar and the camera are abnormally associated, resulting in poor fusion effect, for example, the position, speed, and heading angle of the fused target are updated incorrectly.

[0081] To avoid the above problems, in the embodiment, the perception area is divided into three areas, which are sequentially divided into a target creation area, a data fusion area, and other areas according to the moving direction of the target. The target creation area only obtains visual perception data by the camera, the data fusion area obtains radar perception data by the radar, and the other areas only obtain radar perception data by the radar.

[0082] In the embodiment, the category of the unassociated perception data in the data association result can be determined, for example, the unassociated perception data is visual perception data or radar perception data. If the unassociated perception data is visual perception data in the target creation area or the data fusion area, a new fusion target is created and added to the fusion target list. This process can be understood as that a new fusion target is created when a certain target creation condition is met.

[0083] S260, according to the unassociated fusion target list in the data association result, deleting the expired fusion target in the fusion target list.

[0084] In the embodiment, in addition to updating the fusion target list according to the target association pairs and the unassociated perception data pairs in the data association result, the fusion target list can also be updated based on the unassociated fusion target list in the data association result.

[0085] This step is used to update the unassociated fusion target, that is, to delete the expired fusion target. In the embodiment, after the to-be-fused perception data is obtained, the category of the to-be-fused perception data can be determined. If the to-be-fused perception data is visual perception data, the time length of the target attribute of the fusion target list which is not updated by the radar can be updated, which can also be understood as increasing the time length of the radar which is not updated. If the to-be-fused perception data is radar perception data, the time length of the target attribute of the fusion target list which is not updated by the camera can be updated, which can also be understood as increasing the time length of the camera which is not updated.

[0086] If the perception data of the two categories of the fusion target is not updated, and the time of not updating exceeds the set time threshold, the fusion target can be taken as an expired fusion target, and the expired fusion target is deleted from the fusion target list. For example, a fusion target has driven out of other areas and away from the radar, and the radar and the camera cannot detect the perception data of the fusion target, so that the fusion target has no new perception data for fusion, and when the set time threshold is exceeded, the fusion target can be determined as an expired fusion target, and the expired fusion target is deleted from the fusion target list.

[0087] S270, obtaining the associated perception data of each fusion target in the fusion target list.

[0088] Specifically, the fusion target list can be traversed to obtain the associated perception data associated with each fusion target in the fusion target list.

[0089] S280, converting each associated perception data according to a set coordinate system to obtain fusion target information in a converted perception area.

[0090] The set coordinate system can be set according to actual needs, for example, it can be a latitude and longitude coordinate system. In the embodiment, since each associated perception data is based on a camera coordinate system or a radar coordinate system, after the association is updated, the required latitude and longitude coordinate system is calculated, and the corresponding latitude, ID, heading angle, speed, target type, license plate, vehicle color and other information are output as the fusion target information in the converted perception area.

[0091] The technical scheme provided in the second embodiment of the application specifically determines how to determine the data association result, how to update the fusion target list according to the data association result, and how to determine the fusion target information of the perception area according to the associated perception data of each fusion target in the fusion target list. According to the data association strategy, the data association result of the to-be-fused perception data and the fusion target list is determined, the fusion target list is updated based on different association results in the data association result, the update of the association attribute in the fusion target list is realized, the creation of a new fusion target and the deletion of an expired fusion target are realized, so that accurate data is saved in the target list, and the fusion target information can be output. The accuracy of the target fusion information is ensured, and the target fusion effect is improved. And by setting the capture radar perception area in other areas, since the detection distance of the radar is far, the farthest detection distance of the perception area is no longer limited by the detection distance of the camera, the range of target fusion is increased, and the target fusion effect is further improved.

[0092] As the first optional embodiment of the second embodiment of the present application, the optional embodiment can optimize the implementation of determining the data association result of the to-be-fused perception data and the fusion target list according to the preset data association strategy, and the implementation includes the following steps:

[0093] a1) determining a first association result of the to-be-fused perception data and the fusion target list according to a first data association strategy.

[0094] The first data association strategy refers to association by identification number. First, ID association is performed, and this step of association depends on the tracking of the original sensor. That is, the fusion target is associated with the to-be-fused perception data (radar or camera target) before. In this fusion, as long as the heading angle difference, position difference, and speed difference between the fusion target and the measurement value are within a certain threshold range, the fusion target is associated with the to-be-fused perception data.

[0095] After the first step of association, three kinds of association results are generated, one is a successfully associated target association pair, the other is unassociated to-be-fused perception data, and the other is unassociated fusion target. These results are recorded as the first association result.

[0096] As a specific implementation, the implementation of determining the first association result of the to-be-fused perception data and the fusion target list according to the first data association strategy includes the following steps:

[0097] a11) traversing the to-be-associated identification numbers of the to-be-associated perception targets in the to-be-fused perception data and the associated identification numbers of the associated perception targets of the fusion targets in the fusion target list.

[0098] The targets involved in the to-be-fused perception data are recorded as to-be-associated perception targets, and the identification numbers of the to-be-associated perception targets are recorded as to-be-associated identification numbers. The associated targets of the fusion targets included in the fusion target list are recorded as associated perception targets, and the identification numbers of the associated perception targets are recorded as associated identification numbers. Specifically, the to-be-associated identification numbers of the to-be-associated perception targets in the to-be-fused perception data and the associated identification numbers of the associated perception targets of the fusion targets in the fusion target list are traversed.

[0099] a12) if there is at least one to-be-associated identification number that is the same as the associated identification number, determining whether the to-be-fused perception data meets a preset check condition.

[0100] For example, assuming that the fusion targets in the fusion target list are described by identification numbers 1, 2, and 3, respectively, and that the fusion targets have been associated with perception targets described by identification numbers A, B, and C, respectively, and that 1 has been associated with A, 2 has been associated with B, and 3 has been associated with C, if the newly-arriving perception data to be fused involves an identification number B to be associated, it can be determined through traversal that B is associated with 2, and it is preliminarily considered that the perception data to be fused is also associated with 2.

[0101] However, before association, it is further necessary to determine whether the perception data to be fused satisfies a preset check condition. The purpose of preset checking is that there is a time difference between the data of the previous frame and the newly-arriving data of the current frame, and there is a displacement error due to time when the previous frame and the newly-arriving current frame are associated. If the time interval between the two frames is large, the position difference between the newly-arriving current frame and the previous frame can be very large. Therefore, the trajectory of the previous frame is predicted according to the time difference and the current speed, and the predicted trajectory position and the position of the newly-arriving current frame are compared to determine the position difference. For example, if the position difference between the two is greater than 10 meters, the previous perception data is associated with the fusion target, but the position difference between the current frame and the previous frame is too large, and it is considered that the previous association is incorrect. Therefore, the identification number is forcibly disassociated, and it is considered that the previous association is an incorrect association. If the identification number is associated again, there will be a problem.

[0102] It can be understood that, through the three association strategies, if any step of association is incorrect, the identification number is associated again in the next step, and it is theoretically considered that they are still associated. In fact, the previous step of association is incorrect, and a step of checking is added when the association is performed again. The error between the predicted position, the heading angle, and the speed and the current frame is compared, and the error is checked with a set threshold. It can also be understood that it is necessary to satisfy a condition to consider that the identification number associated previously is still associated. However, when the distance threshold is determined, the position of the previous frame is predicted. Since the newly-arriving current frame and the previous frame are not at the same time, the time difference between the previous frame and the current frame is predicted according to uniform linear motion or uniform accelerated linear motion, and the position difference between the predicted position and the newly-arriving position is calculated.

[0103] For example, it is assumed that a fusion target and a perception target have been associated before, for example, the fusion target is straight, and the perception target is parallel to the fusion target but turns right, and then the perception data of the perception target is associated with the fusion target due to the close distance between the two, and at the next moment, the fusion target actually moves straight forward, and the perception target turns right. Since the two have been associated before, they are still considered to be associated at this time, which causes association error. Considering that the distance between the two becomes larger and larger as time goes back, the preset verification condition can be used to solve the problem of misassociation. In other words, the fusion target and the to-be-associated perception data that meet the preset verification condition are associated, and the to-be-associated perception data that do not meet the verification condition are not associated, which avoids misassociation and reduces the amount of calculation.

[0104] If the above condition is met, it is necessary to continue to determine whether the unassociated perception data meets the preset verification condition. First, the time difference between the first unassociated perception data of the current frame and the front and back frames of the associated data of the fusion target in the last frame is calculated. According to the time difference, the position, speed and heading angle of the associated data in the last frame, the position, speed and heading angle of the fusion target in the current frame can be predicted. Further, the position difference, speed difference and heading angle difference between the current frame and the predicted position can be calculated. It should be noted that if the speed in the first unassociated perception data is very small, at this time, the heading angle is considered to be locked, and only the position difference and speed difference between the current frame and the predicted position need to be calculated. The preset verification condition refers to that the position difference, speed difference and heading angle difference are within the set threshold range. For example, it is assumed that the fusion target is a uniform straight line motion, and the position of the target in the current frame can be predicted according to the time difference and the speed.

[0105] a13) If the condition is met, the fusion target associated with the associated identification number with the same identification number and the to-be-associated perception target are determined as the first target association pair.

[0106] Specifically, if a to-be-associated identification number is the same as an associated identification number, and the to-be-associated perception data of the fusion target associated with the specific associated identification number and the to-be-associated perception target are considered as a target association pair, which is recorded as the first target association pair.

[0107] a14) The perception data in the to-be-associated perception data except the to-be-associated perception target in the first target association pair is taken as the first unassociated perception data.

[0108] Specifically, the perception data in the to-be-associated perception data except the to-be-associated perception data of the target association pair is taken as the unassociated perception data, which is recorded as the first unassociated perception data.

[0109] a15) taking the fusion targets in the fusion target list other than the fusion target in the first target association pair as first unassociated fusion targets.

[0110] Specifically, the fusion targets in the fusion target list other than the fusion target in the above target association pair are taken as unassociated fusion targets, which are recorded as first unassociated fusion targets.

[0111] a16) taking the first target association pair, the first unassociated perception data, and the first unassociated fusion target as a first association result.

[0112] The above technical solution specifically implements the step of associating the to-be-fused perception data and the fusion target list by the identification number to obtain a first association result.

[0113] b1) if the first association result contains the first unassociated perception data and the first unassociated fusion target, determining a second association result of the first unassociated perception data and the first unassociated fusion target according to a second data association strategy.

[0114] Specifically, if there are still unassociated to-be-fused perception data and unassociated fusion targets after the previous data association strategy, a second association is performed. For example, a target moves from a target creation area to a data fusion area. When moving to the data fusion area for the first time, there may be radar perception data without identification number association. At this time, only IOU can be performed. When performing IOU association, it is necessary to determine whether at least one data belongs to the data fusion area, which can also be understood as performing IOU association only when at least one data belongs to the data fusion area. When the fusion target moves to the data fusion area, if the incoming to-be-fused perception data is radar perception data, the visual perception data associated in the fusion target is extracted and the intersection over union of the frames is calculated with the new radar perception data. Similarly, if the incoming to-be-fused data is visual fusion data, the radar perception data associated in the fusion target is extracted and the intersection over union of the frames is calculated with the new visual fusion data to generate a matching cost matrix, which is input into Hungarian matching to obtain a data association result.

[0115] In this embodiment, the first unassociated perception data and the first unassociated fusion target are associated by IOU, and the result after association is recorded as a second association result. It can be known that the second association result can contain a target association pair, which is recorded as a second target association pair; can also contain unassociated to-be-fused perception data, which is recorded as second unassociated perception data; and can also contain unassociated fusion targets, which are recorded as second unassociated fusion targets.

[0116] As a specific implementation, the implementation of determining the second association result of the first unassociated perception data and the first unassociated fusion target according to the second data association strategy can include the following steps:

[0117] b11) traversing the first unassociated perception data and the first unassociated fusion target.

[0118] b12) determining whether there is at least one frame of first unassociated perception data or data in the first unassociated fusion target belonging to the data fusion region.

[0119] Specifically, determining whether there is data in a frame of first unassociated perception data or the first unassociated fusion target involved in the data fusion region can also be understood as corresponding data being data captured when the target is in the data fusion region.

[0120] If the above conditions are not met, the determination of the fusion target in the first unassociated perception data and the first unassociated fusion target is not associated. At this time, the minimum intersection over union can be set to make the result output by the subsequent Hungarian algorithm be no association.

[0121] b13) if yes, determining whether the first unassociated perception data meets a preset check condition.

[0122] If the above condition is met, it is necessary to continue to determine whether the unassociated perception data meets the preset check condition. First, the time difference between the current frame of the first unassociated perception data and the previous and next frames of the associated data of the fusion target in the last frame is calculated. According to the time difference, the position, speed and heading angle of the last frame of associated data, the position, speed and heading angle of the current frame of the fusion target can be predicted. Further, the position difference, speed difference and heading angle difference between the current frame and the predicted position can be calculated. It should be noted that if the speed in the first unassociated perception data is very small, at this time, the heading angle is considered to be locked, and only the position difference and speed difference between the current frame and the predicted position need to be calculated. The preset check condition refers to that the position difference, speed difference and heading angle difference are within the set threshold range. The purpose of the check is described above and will not be discussed here.

[0123] b14) if yes, performing intersection over union calculation on the heterogeneous perception data in the first unassociated perception data and the first unassociated fusion target to determine the first association matrix.

[0124] If the preset check condition is met, the IOU calculation is performed on the heterogeneous perception data in the first associated perception data and the first unassociated fusion target, and a matrix generated according to (1-IOU) is denoted as the first association matrix.

[0125] The heterogeneous perception data in the first unassociated fusion target refers to radar perception data if the first unassociated perception data is visual perception data. The heterogeneous perception data in the first unassociated fusion target refers to visual perception data if the first unassociated perception data is radar perception data.

[0126] b15) inputting the first association matrix into the Hungarian algorithm to obtain a second target association pair, second unassociated perception data, and a second unassociated fusion target, and taking the second target association pair, the second unassociated perception data, and the second unassociated fusion target as a second association result.

[0127] Specifically, the first association matrix is input into the Hungarian algorithm, and the Hungarian algorithm is used for association matching to determine whether the first unassociated perception data can be associated with the first unassociated fusion target. In this embodiment, the target association pair obtained through this step is referred to as a second target association pair, the unassociated perception data obtained is referred to as second unassociated perception data, and the second unassociated fusion target obtained is referred to as a second unassociated fusion target. The second target association pair, the second unassociated perception data, and the second unassociated fusion target are taken as a second association result.

[0128] The above technical solution specifically realizes the step of associating the first unassociated perception data with the first unassociated fusion target by the IOU association to obtain a second association result.

[0129] c1) if the second association result contains the second unassociated perception data and the second unassociated fusion target, determining a third association result of the second unassociated perception data and the second unassociated fusion target according to a third data association strategy.

[0130] Specifically, if there are still unassociated perception data to be fused and unassociated fusion targets after the previous two data association strategies, the unassociated perception data to be fused is referred to as second unassociated perception data, and the unassociated fusion target is referred to as a second unassociated fusion target in this embodiment. In this step, distance association is performed on the previous two unassociated fusion target lists, which is referred to as a third data association strategy. It can be understood that when the fusion target moves from the fusion area to other areas, the fusion target has carried the unique properties of the camera (such as the license plate, type, and color of the vehicle) and the advantages of the radar (such as the position, speed, and heading angle of the vehicle). Therefore, the fusion target in the next area increases distance association, and the radar continues to maintain the position, speed, and heading angle of the target.

[0131] In this embodiment, the second unassociated perception data and the second unassociated fusion target are distance associated, and the associated result is recorded as a third associated result. It can be known that the third associated result can contain target associated pairs, recorded as a third target associated pair; can also contain unassociated perception data to be fused, recorded as third unassociated perception data; and can also contain unassociated fusion targets, recorded as third unassociated fusion targets.

[0132] As a specific implementation, the implementation of determining the third associated result of the second unassociated perception data and the second unassociated fusion target according to the third data association strategy can include the following steps:

[0133] c11) traversing the second unassociated perception data and the second unassociated fusion target.

[0134] c12) judging whether the data in the second unassociated perception data and the second unassociated fusion target are all in other areas and the second unassociated perception data is radar perception data.

[0135] Specifically, it is judged whether the data involved in the second unassociated perception data and the second unassociated fusion target are all in other areas, which can also be understood as the corresponding data being data captured when the target is in other areas. It is also needed to judge whether the second unassociated perception data is radar perception data.

[0136] If the above conditions are not met, it can be determined that the second unassociated perception data and the second unassociated fusion target are not associated. At this time, the amplitude of the cost element can be set to be maximum, so that the result output by the subsequent Hungarian algorithm is not associated.

[0137] c13) if yes, judging whether the second unassociated perception data satisfies a preset check condition.

[0138] If the above conditions are met, it is needed to continue to judge whether the unassociated perception data satisfies the preset check condition. Firstly, the time difference between the second unassociated perception data of the current frame and the associated data of the fusion target of the previous frame is calculated. According to the time difference, the position, speed and heading angle of the previous frame associated data, the position, speed and heading angle of the current frame fusion target can be predicted. Further, the position difference, speed difference and heading angle difference between the current frame and the predicted position can be calculated. It should be noted that if the speed in the second unassociated perception data is very small, at this time, the heading angle is considered to be locked, and only the position difference and speed difference between the current frame and the predicted position need to be calculated. The preset check condition refers to that the position difference, speed difference and heading angle difference are within the set threshold range. The check purpose is described above and will not be discussed here.

[0139] c14) If the preset check condition is met, distance calculation is performed on the second unassociated perception data and the same type of perception data in the second unassociated fusion target to determine a second association matrix.

[0140] The second association matrix is a matrix indicating the association relationship between the second unassociated perception data and the fusion target in the second unassociated fusion target. Specifically, if the preset check condition is met, distance calculation is performed on the second unassociated perception data and the same type of perception data in the second unassociated fusion target to determine a second association matrix. The specific distance calculation method is: straight-line distance / (threshold distance*2).

[0141] The same type of perception data in the second unassociated perception data and the second unassociated fusion target refers to, if the second unassociated perception data is visual perception data, the same type of perception data in the second unassociated fusion target refers to visual perception data. If the second unassociated perception data is radar perception data, the same type of perception data in the second unassociated fusion target refers to radar perception data.

[0142] c15) The second association matrix is input into the Hungarian algorithm to obtain a third target association pair, third position associated perception data, and a third unassociated fusion target, and the third target association pair, the third unassociated perception data, and the third unassociated fusion target are taken as a third association result.

[0143] Specifically, the second association matrix is input into the Hungarian algorithm to perform association matching through the Hungarian algorithm to determine whether the second unassociated perception data can be associated with the second unassociated fusion target. In this embodiment, the target association pair obtained through this step is referred to as a third target association pair, the unassociated perception data obtained is referred to as third unassociated perception data, and the third unassociated fusion target obtained is referred to as a third unassociated fusion target. The third target association pair, the third unassociated perception data, and the third unassociated fusion target are taken as a third association result.

[0144] The above technical solution specifically realizes the step of associating the second unassociated perception data and the second unassociated fusion target through distance association to obtain a third association result.

[0145] d1) According to the first association result, the second association result, and the third association result, a data association result is determined.

[0146] The first association result contains a first target association pair, first unassociated perception data, and a first unassociated fusion target. The first target association pair is obtained from the first association result, which is equivalent to obtaining a target association pair associated based on a first data association strategy. The second association result contains a second target association pair, second unassociated perception data, and a second unassociated fusion target. The second target association pair is obtained from the second association result, which is equivalent to obtaining a target association pair associated based on a second data association strategy. The third association result contains a third target association pair, third unassociated perception data, and a third unassociated fusion target. The third target association pair, the third unassociated perception data, and the third unassociated fusion target are obtained.

[0147] When the first association result, the second association result, and the third association result are obtained, a final data association result after three associations is determined according to the results contained in the first association result, the second association result, and the third association result.

[0148] The above technical solution specifically determines the step of determining the data association result of the perception data to be fused and the fusion target list according to the preset data association strategy. Through three data association strategies, the determination of the data association result is realized, which provides a basis for subsequent attribute updating of the fusion target.

[0149] As a specific implementation, the implementation of determining the data association result according to the first association result, the second association result, and the third association result includes the following steps:

[0150] d11) obtaining the first target association pair in the first association result, the second target association pair in the second association result, the third target association pair, the third unassociated perception data, and the third unassociated fusion target in the third association result.

[0151] The first association result contains a first target association pair, first unassociated perception data, and a first unassociated fusion target. The first target association pair is obtained from the first association result, which is equivalent to obtaining a target association pair associated based on a first data association strategy. The second association result contains a second target association pair, second unassociated perception data, and a second unassociated fusion target. The second target association pair is obtained from the second association result, which is equivalent to obtaining a target association pair associated based on a second data association strategy. The third association result contains a third target association pair, third unassociated perception data, and a third unassociated fusion target. The third target association pair, the third unassociated perception data, and the third unassociated fusion target are obtained.

[0152] d12) taking the first target association pair, the second target association pair, and the third target association pair as a target association pair in the data association result.

[0153] It can be understood that the result generated after data association can contain three kinds: target association pairs on association, perception data on non-association, and fusion target on non-association. This step is used to determine the target association pairs on association.

[0154] In this embodiment, the first target association pair corresponds to the target association pair on association based on the first data association strategy, the second target association pair corresponds to the target association pair on association based on the second data association strategy, and the third target association pair corresponds to the target association pair on association based on the third data association strategy. The target association pairs obtained by the three times of association can be taken as the target pairs on all associations in this round, that is, the first target association pair, the second target association pair, and the third target association pair can be taken as the target association pairs in the data association result.

[0155] d13) taking the third non-associated perception data as the non-associated perception data in the data association result.

[0156] Among them, the third non-associated perception data can be considered as the to-be-fused perception data on which the three times of data association strategies are not associated, and in this embodiment, the third non-associated perception data can be taken as the non-associated perception data in the data association result.

[0157] d14) taking the third non-associated fusion target as the non-associated fusion target in the data association result.

[0158] Among them, the third non-associated fusion target can be considered as the fusion target on which the three times of data association strategies are not associated, and in this embodiment, the third non-associated fusion target can be taken as the non-associated fusion target in the data association result.

[0159] The above technical solution specifically implements the steps of how to determine the data association result according to the first association result, the second association result, and the third association result, extracts the association pairs in the three times of association results as the target association pairs, takes the third non-associated perception data on which the three times of data association strategies are not associated as the non-associated perception data in the data association result, and takes the third non-associated fusion target on which the three times of data association strategies are not associated as the non-associated fusion target in the data association result. This provides basic data for subsequent updating of the fusion target list.

[0160] As a second optional embodiment of the second embodiment of the present application, the optional embodiment can optimize the updating of the target attribute of the fusion target association in the fusion target list according to the target association pairs in the data association result on the basis of the above-mentioned embodiment, which includes:

[0161] a2) determining the to-be-added target attribute according to the to-be-fused perception data corresponding to the target association pair.

[0162] In this embodiment, after the target association pair is determined, the attributes of the target association pair need to be updated, that is, the attributes of the fusion target corresponding to the to-be-fused perception data are determined according to the to-be-fused perception data corresponding to the target association pair, and are denoted as to-be-added target attributes.

[0163] It can be understood that the to-be-fused perception data corresponding to the target association pair can be visual perception data or radar perception data. If the to-be-fused perception data corresponding to the target association pair is visual perception data, the static attributes in the visual perception data are used as the to-be-added target attributes, and since some dynamic attributes such as the position, speed, heading angle, and pixel coordinates of the target vehicle can also be determined according to the data captured by the camera, these dynamic attributes need to be fused with the radar perception data.

[0164] If the to-be-fused perception data corresponding to the target association pair is radar perception data, the dynamic attributes in the radar perception data are used as the to-be-added target attributes. At the same time, since some dynamic attributes can also be determined according to the data captured by the camera, the dynamic attributes corresponding to the radar perception data and the dynamic attributes corresponding to the associated visual perception data can be fused.

[0165] As a specific implementation, the implementation of determining the to-be-added target attributes according to the to-be-fused perception data corresponding to the target association pair in the data association result can be specifically optimized as the following steps:

[0166] a21) If the to-be-fused perception data corresponding to the target association pair is visual perception data, the first proportion value of the first perception data in the to-be-fused perception data is determined according to the association time length of the to-be-fused perception data.

[0167] It is considered that after the target moves from the target creation region to the data fusion region, the attributes of the target in the target creation region are provided by the camera, and after moving to the data fusion region, the target is associated with the radar, so that the position information of the target has the position information captured by the radar. At this time, the position information captured by the camera and the radar can have a position difference. When running from the target creation region to the data fusion region, a position jump can occur. Based on this, in this embodiment, when the to-be-fused perception data is visual perception data, the association time length of the fusion target associated with the camera is updated, and based on the association time length, the position information, speed, heading angle, and error proportion weight corresponding to the visual perception data can be determined. In this embodiment, the dynamic attributes such as the position information, speed, and heading angle corresponding to the visual perception data are denoted as first perception data, and the proportion weight corresponding to the first perception data is denoted as the first proportion value of the first perception data. It can be known that the first proportion value is calculated according to the first perception data and the radar perception data associated with the fusion target.

[0168] It needs to be clear that the purpose of determining the proportion value of the radar in the embodiment by updating the association duration of the radar associated with the fusion target is to retain a part of the dynamic attributes determined by the radar, so that the dynamic attributes are not directly changed from the values determined by the camera to the values determined by the radar. Instead, these dynamic attributes are jointly maintained by the camera position and the radar position to slowly move forward, and move forward until only the radar captures data, at which time all information is only radar data. This process can be understood as a trajectory smoothing operation.

[0169] a22) determining a first target attribute according to the first proportion value and the first perception data, and taking the first target attribute and the second perception data in the perception data to be fused as the target attribute to be added.

[0170] In the embodiment, when the first perception data, the radar perception data associated with the fusion target corresponding to the target association pair, and the first proportion value of the first perception data are known, the dynamic attributes such as position information, speed, and heading angle can be updated to obtain updated target attributes, denoted as first target attributes. At the same time, the static attributes related to the fusion target, such as the type, color, and license plate of the vehicle, are only determined by the camera, and the determined information is relatively accurate, so the static attributes in the perception data to be fused can be taken as the static attributes of the fusion target. In the embodiment, the static attributes in the perception data to be fused are taken as second perception data, and the first target attributes and the second perception data in the perception data to be fused are taken as the target attributes to be added.

[0171] a23) If the perception data to be fused corresponding to the target association pair is radar perception data, determining a second proportion value of third perception data in the perception data to be fused according to the association duration of the perception data to be fused.

[0172] In the embodiment, if the perception data to be fused corresponding to the target association pair is radar perception data, i.e., it involves the dynamic attributes of the fusion target, in order to ensure the trajectory smoothing of the fusion target, the proportion of the perception data to be fused is also used to update the target attributes to be added.

[0173] Specifically, the purpose of determining the proportion value of the radar by updating the association duration of the radar associated with the fusion target is to retain the dynamic attributes determined by the radar, so that the dynamic attributes are not directly changed from the values determined by the camera to the values determined by the radar. Instead, these dynamic attributes are jointly maintained by the camera position and the radar position to slowly move forward, and move forward until only the radar captures data, at which time all information is only radar data. This process can be understood as a trajectory smoothing operation.

[0174] Based on this, in the embodiment, when the to-be-fused perception data is radar perception data, the association duration of the radar associated with the fusion target is updated, and based on the association duration, the position information, speed, heading angle and error proportion weight corresponding to the radar perception data can be determined. In the embodiment, the position information, speed, heading angle and other dynamic attributes corresponding to the radar perception data are recorded as third perception data, and the proportion weight corresponding to the third perception data is recorded as a second proportion value of the third perception data. It can be known that the proportion value here is calculated according to the third perception data and the visual perception data associated with the fusion target.

[0175] a24) determining a third target attribute according to the second proportion value and the third perception data, and taking the third target attribute as the to-be-added target attribute.

[0176] In the embodiment, when the third perception data, the visual perception data associated with the fusion target corresponding to the target association pair, and the proportion value of the third perception data are known, the dynamic attributes such as position information, speed, heading angle, etc. can be updated to obtain updated target attributes, which are recorded as third target attributes. The third target attribute is taken as the to-be-added target attribute.

[0177] The above technical solution specifically realizes the determination process of the to-be-added target attribute. For different categories of to-be-fused perception data, different ways are used to determine the to-be-added target attribute, and the proportion values of the visual perception data and the radar perception data are used to realize the smoothing processing of the fusion target moving track, so that the determined to-be-added target attribute is more accurate and real.

[0178] b2) determining the fusion target corresponding to the target association pair from the fusion target list as the to-be-added target.

[0179] Specifically, the fusion target corresponding to the target association pair is determined from the fusion target list, and the fusion target is taken as the to-be-added target.

[0180] c2) adding the to-be-added target attribute to the to-be-added target.

[0181] Specifically, the to-be-added target attribute can be associated with the to-be-added target to realize the addition of the to-be-added target attribute to the to-be-added target.

[0182] The above technical solution specifically realizes the step of updating the target attribute of the fusion target associated in the fusion target list according to the target association pair in the data association result, realizes the instant update of the fusion target attribute, and guarantees the accuracy of the target attribute.

[0183] As a third optional embodiment of the second embodiment of the present application, the optional embodiment can optimize the unassociated perception data in the data association result, create a new fusion target and store it in the fusion target list, including:

[0184] a3) determining the category of the unassociated perception data.

[0185] In this embodiment, the category of the unassociated perception data in the data association result can be determined, for example, the unassociated perception data is visual perception data or radar perception data.

[0186] b3) if the unassociated perception data is visual perception data of the target creation area or the data fusion area, a new fusion target is created and added to the fusion target list.

[0187] In the prior art, visual perception data captured by a camera is associated with radar perception data captured by a radar in the entire perception area. Since the radar is prone to produce multiple clustering points for a target with a large volume, for example, a large truck produces three clustering points, and is detected as three targets, two of which are false targets, causing radar false detection, missed detection, etc. When the two false targets are mapped to the image captured by the camera, the radar and the camera will be abnormally associated, resulting in poor fusion effect, for example, the position, speed, heading angle, etc. of the fusion target are updated incorrectly.

[0188] In order to avoid the false detection and missed detection problem of the radar, in this embodiment, only the visual perception data captured by the camera is used for fusion target creation when creating a target, that is, a corresponding fusion target is created based on the target identified in the visual perception data. The visual perception data of the target creation area or the data fusion area with high camera detection accuracy is used for fusion target creation, and the newly created fusion target is added to the fusion target list.

[0189] c3) adding the unassociated perception data as a target attribute to the new fusion target.

[0190] After creating a new fusion target in the fusion target list, the attributes contained in the unassociated perception data can be added to the new fusion target as target attributes. It can be understood that the added target attributes are based on visual perception data, for example, assuming that the fusion target is a vehicle, the target attributes include target type, vehicle type, license plate, vehicle color, etc.

[0191] Of course, it can also include vehicle position, vehicle speed, vehicle pixel coordinates, and heading angle, etc. However, these attributes are determined based on the image captured by the camera, and the error is larger than the attributes determined by the radar.

[0192] d3) If the unassociated perception data is radar perception data, the fusion target list is not updated.

[0193] Specifically, since there is a possibility of missing detection and false detection of the target by the radar, target creation is only based on the visual perception data captured by the camera, and target creation is not based on the radar perception data. Therefore, if the unassociated perception data is radar perception data, a new fusion target is not created, that is, the fusion target list is not updated.

[0194] The above technical solution specifically realizes how to create a new fusion target according to the unassociated perception data in the data association result and store it in the fusion target list, and ensures the accuracy of the created fusion target by creating the fusion target based on the visual perception data captured by the target creation area or the data fusion area.

[0195] As a fourth optional embodiment of the second embodiment of the present application, the fourth optional embodiment can optimize the unassociated fusion target list according to the data association result and delete the expired fusion target in the fusion target list based on the above-mentioned embodiments, comprising:

[0196] a4) If the unassociated fusion target is not associated for a time greater than a set time threshold, the unassociated fusion target is determined to be an expired fusion target.

[0197] In this embodiment, after the to-be-fused perception data is obtained, the category of the to-be-fused perception data can be determined. If the to-be-fused perception data is visual perception data, the time length of the radar not participating in updating the target attribute of the fusion target list can be updated, which can also be understood as increasing the time length of the radar not participating in updating. If the to-be-fused perception data is radar perception data, the time length of the camera not participating in updating the target attribute of the fusion target list can be updated, which can also be understood as increasing the time length of the camera not participating in updating.

[0198] Wherein, the fusion target in the unassociated fusion target list is marked as an unassociated fusion target. The unassociated fusion target is traversed, and if the time length of the radar not participating in the target attribute under a certain unassociated fusion target exceeds a set time threshold, the radar attribute of invalid association can be set. If the time length of the camera not participating in the target attribute under a certain unassociated fusion target exceeds a set time threshold, the camera attribute of invalid association can be set. When the unassociated fusion target is not associated for a time greater than a set time threshold, it is equivalent to that there is no valid associated radar attribute or valid associated camera attribute under the unassociated fusion target, and the unassociated fusion target is determined to be an expired fusion target.

[0199] b4) Delete the expired fusion target from the fusion target list.

[0200] Specifically, if it is determined that the expired fusion target, the expired fusion target can be deleted from the fusion target list.

[0201] The technical solution above specifically realizes how to delete the expired fusion target in the fusion target list according to the unassociated fusion target in the data association result. The expired fusion target is determined by the time length of non-participation in updating, and the expired fusion target is deleted, which saves resource occupation, improves the speed of traversing the fusion target list, and further improves the efficiency of data fusion.

[0202] Embodiment three

[0203] Figure 4 A structural schematic diagram of a roadside perception device based on multiple sensors is provided for the third embodiment of the present application. The device can be applied to the case of roadside perception fusion based on visual sensors and millimeter wave radars. The roadside perception device based on multiple sensors can be configured in an edge computing device, as shown in the figure. The device comprises a data acquisition module 31, a target association module 32, a data fusion module 33 and an information determination module 34; wherein, Figure 4

[0204] The data acquisition module 31 is configured to acquire the to-be-fused perception data of the perception area.

[0205] The target association module 32 is configured to determine the data association result of the to-be-fused perception data and the fusion target list according to the to-be-fused perception data and the pre-created fusion target list.

[0206] The data fusion module 33 is configured to update the fusion target list according to the data association result.

[0207] The information determination module 34 is configured to determine the fusion target information of the perception area according to the associated perception data corresponding to each fusion target in the fusion target list.

[0208] The perception area is divided into a target creation area, a data fusion area and other areas in turn according to the target moving direction. The target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other areas correspond to radar perception data.

[0209] ​The above technical solution divides the perception area into three regions based on the detection accuracy of different sensors. In the target creation region, only visual perception data is captured; in the data fusion region, both visual perception data and radar perception data are captured; and in other regions, only radar perception data is captured. This means that the fusion of visual perception data and radar perception data only begins in the data fusion region where both camera and radar detection accuracy are high. This avoids anomalies in radar and camera correlation, ensuring the accuracy of target fusion information and improving the target fusion effect. Furthermore, by setting up radar perception regions in other areas, the maximum detection distance of the perception area is no longer limited by the camera's detection distance due to the longer radar detection range, increasing the scope of target fusion and further improving the target fusion effect.

[0210] Optionally, the target association module 32 may specifically include:

[0211] The first association unit is used to treat the unassociated sensing data to be fused as the data association result between the sensing data to be fused and the fusion target list if the fusion target list is empty.

[0212] The second association unit is used to determine the data association result between the sensing data to be fused and the fusion target list according to a preset data association strategy if the fusion target list is not empty.

[0213] Optionally, the second associated unit may specifically include:

[0214] The first determining subunit is used to determine the first association result between the sensing data to be fused and the fusion target list according to the first data association strategy;

[0215] The second determining subunit is used to determine the second association result of the first unassociated sensing data and the first unassociated fusion target according to the second data association strategy if the first association result contains the first unassociated sensing data and the first unassociated fusion target.

[0216] The third determining subunit is used to determine the third association result between the second unassociated sensing data and the second unassociated fusion target according to the third data association strategy if the second association result contains the second unassociated sensing data and the second unassociated fusion target.

[0217] The fourth determining subunit is used to determine the data association result based on the first association result, the second association result, and the third association result.

[0218] Optionally, the first determined sub-unit can be specifically used for:

[0219] Iterate through the identification numbers of the sensing targets to be associated in the sensing data to be fused and the identification numbers of the sensing targets already associated in the fusion target list for each fusion target;

[0220] If there is at least one to-be-associated identification number same as the associated identification number, it is judged whether the to-be-fused perception data meets a preset check condition;

[0221] If yes, it is determined that the fusion target associated with the associated identification number with the same identification number and the to-be-associated perception target are a first target association pair;

[0222] The perception data in the to-be-fused perception data except the to-be-associated perception target in the first target association pair is taken as first unassociated perception data;

[0223] The fusion target in the fusion target list except the fusion target in the first target association pair is taken as first unassociated fusion target;

[0224] The first target association pair, the first unassociated perception data and the first unassociated fusion target are taken as a first association result.

[0225] Optionally, the second determination subunit can be specifically used for:

[0226] Iterating the first unassociated perception data and the first unassociated fusion target;

[0227] It is judged whether there is at least one frame of data in the first unassociated perception data or the first unassociated fusion target belonging to the data fusion region;

[0228] If yes, it is judged whether the first unassociated perception data meets a preset check condition;

[0229] If yes, the first unassociated perception data and the first unassociated fusion target are calculated by using the intersection over union method to determine a first association matrix;

[0230] The first association matrix is input into the Hungarian algorithm to obtain a second target association pair, second unassociated perception data and second unassociated fusion target, and the second target association pair, the second unassociated perception data and the second unassociated fusion target are taken as a second association result.

[0231] Optionally, the third determination subunit can be specifically used for:

[0232] Iterating the second unassociated perception data and the second unassociated fusion target;

[0233] It is judged whether the data in the second unassociated perception data and the second unassociated fusion target all belong to other regions and the second unassociated perception data is radar perception data;

[0234] If yes, it is judged whether the second unassociated perception data meets a preset check condition;

[0235] If yes, distance calculation is performed on the second unassociated perception data and the same type of perception data in the second unassociated fusion target to determine a second association matrix;

[0236] The second association matrix is input into a Hungarian algorithm to obtain a third target association pair, third unassociated perception data and a third unassociated fusion target, and the third target association pair, the third unassociated perception data and the third unassociated fusion target are taken as a third association result.

[0237] Optionally, the fourth determination subunit can be specifically configured to:

[0238] The first target association pair in the first association result, the second target association pair in the second association result, the third target association pair, the third unassociated perception data and the third unassociated fusion target in the third association result are obtained.

[0239] The first target association pair, the second target association pair and the third target association pair are taken as a target association pair in a data association result.

[0240] The third unassociated perception data is taken as unassociated perception data in the data association result.

[0241] The third unassociated fusion target is taken as unassociated fusion target in the data association result.

[0242] Optionally, the data fusion module 33 can specifically include:

[0243] The attribute adding unit is configured to update a target attribute associated with a fusion target in a fusion target list according to a target association pair in the data association result.

[0244] The target creating unit is configured to create a new fusion target according to unassociated perception data in the data association result and store the new fusion target into the fusion target list.

[0245] The target deleting unit is configured to delete an expired fusion target in the fusion target list according to unassociated fusion target in the data association result.

[0246] Optionally, the attribute adding unit can specifically include:

[0247] The attribute determination subunit is configured to determine a target attribute to be added according to to-be-fused perception data corresponding to the target association pair.

[0248] The target determination subunit is configured to determine a fusion target corresponding to the target association pair as a target to be added from the fusion target list.

[0249] The adding subunit is configured to add the target attribute to be added to the target to be added.

[0250] Optionally, the attribute determining subunit can be specifically used for:

[0251] If the target association pair corresponds to visual perception data, a first proportion value of first perception data in the to-be-fused perception data is determined according to an association duration of the to-be-fused perception data;

[0252] A first target attribute is determined according to the first proportion value and the first perception data, and the first target attribute and second perception data in the to-be-fused perception data are used as a to-be-added target attribute;

[0253] If the target association pair corresponds to radar perception target data, a second proportion value of third perception data in the to-be-fused perception data is determined according to an association duration of the to-be-fused perception data;

[0254] A third target attribute is determined according to the second proportion value and the third perception data, and the third target attribute is used as the to-be-added target attribute.

[0255] Optionally, the target creating unit can be specifically used for:

[0256] The category of the unassociated perception data is determined;

[0257] If the unassociated perception data is visual perception data in the target creating area or the data fusion area, a new fusion target is created and stored in the fusion target list;

[0258] The unassociated perception data is added to the new fusion target as a target attribute;

[0259] If the unassociated perception data is radar perception target data, the fusion target list is not updated.

[0260] Optionally, the target deleting unit can be specifically used for:

[0261] If the unassociated fusion target does not participate in association for more than a set time threshold, the unassociated fusion target is determined to be an expired fusion target;

[0262] The expired fusion target is deleted from the fusion target list.

[0263] Optionally, the information determining module 34 can be specifically used for:

[0264] The associated perception data of each fusion target in the fusion target list is obtained;

[0265] Each associated perception data is converted according to a set coordinate system to obtain fusion target information in a converted perception area.

[0266] The multi-sensor-based roadside perception device provided by the embodiment of the present application can execute the multi-sensor-based roadside perception method provided by any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0267] Embodiment four

[0268] Figure 5 A structural schematic diagram of an edge computing device provided by the fourth embodiment of the present application. The edge computing device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The edge computing device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0269] As shown in Figure 5 The edge computing device 40 includes at least one processor 41, and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., which is communicatively connected to the at least one processor 41, wherein the memory stores a computer program executable by the at least one processor. The processor 41 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or loaded into the random access memory (RAM) 43 from the storage unit 48. In the RAM 43, various programs and data required for the operation of the edge computing device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0270] A plurality of components in the edge computing device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the edge computing device 40 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0271] The processor 41 can be various general and / or special purpose processing components having processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 41 performs various methods and processes described above, such as the multi-sensor based roadside perception method.

[0272] In some embodiments, the multi-sensor based roadside perception method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed onto the edge computing device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded onto the RAM 43 and executed by the processor 41, one or more steps of the multi-sensor based roadside perception method described above can be performed. Alternatively, in other embodiments, the processor 41 can be configured to perform the multi-sensor based roadside perception method by any other suitable means, such as by means of firmware.

[0273] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0274] Computer programs used to implement the methods of the application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0275] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0276] To provide for interaction with a user, the systems and techniques described here can be implemented on a computing device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computing device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0277] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0278] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0279] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0280] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-sensor based roadside perception method, characterized in that, The method comprises the following steps: acquiring to-be-fused perception data of a perception area; determining a data association result of the to-be-fused perception data and a pre-created fusion target list according to the to-be-fused perception data and the fusion target list; updating the fusion target list according to the data association result; determining fusion target information of the perception area according to associated perception data corresponding to each fusion target in the fusion target list; the perception area is divided into a target creation area, a data fusion area and other areas in turn according to a target moving direction, the target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other areas correspond to radar perception data; the target creation area is a near-distance area with camera detection accuracy higher than radar detection accuracy, and is used for identifying a target and maintaining static attribute information of a fusion target only through visual perception data captured by a camera; the data fusion area is a middle-distance area with both camera and radar satisfying detection accuracy, and is used for maintaining dynamic attribute information of a fusion target through visual perception data captured by a camera and radar perception data captured by a radar; the other areas are far-distance areas with radar detection accuracy higher than camera detection accuracy, and are used for maintaining dynamic attribute information of a fusion target only through radar perception data captured by a radar; wherein, the determining of the data association result of the to-be-fused perception data and the fusion target list according to the to-be-fused perception data and the pre-created fusion target list comprises: if the fusion target list is not empty, determining the data association result of the to-be-fused perception data and the fusion target list according to a preset data association strategy; wherein, the determining of the data association result of the to-be-fused perception data and the fusion target list according to the preset data association strategy comprises: determining a first association result of the to-be-fused perception data and the fusion target list according to a first data association strategy; if the first association result contains first unassociated perception data and a first unassociated fusion target, determining a second association result of the first unassociated perception data and the first unassociated fusion target according to a second data association strategy; if the second association result contains second unassociated perception data and a second unassociated fusion target, determining a third association result of the second unassociated perception data and the second unassociated fusion target according to a third data association strategy; determining the data association result according to the first association result, the second association result and the third association result; the first data association strategy refers to association through an identification number, the second data association strategy refers to association through an intersection-over-union ratio, and the third data association strategy refers to association through distance.

2. The method of claim 1, wherein, the determining of the data association result of the to-be-fused perception data and the fusion target list according to the to-be-fused perception data and the pre-created fusion target list comprises: if the fusion target list is empty, regarding the to-be-fused perception data as unassociated perception data as the data association result of the to-be-fused perception data and the fusion target list.

3. The method of claim 1, wherein, The first association result of the to-be-fused perception data and the fusion target list is determined according to a first data association strategy, and the first association result comprises the following steps. Traverse the to-be-associated identification number of each to-be-associated perception target in the to-be-fused perception data and the associated identification number of each associated perception target of each fusion target in the fusion target list. If there is at least one to-be-associated identification number identical to the associated identification number, determine whether the to-be-fused perception data satisfies a preset check condition. If yes, determine that the fusion target associated with the associated identification number with the same identification number and the to-be-associated perception target are a first target association pair. Take the perception data in the to-be-fused perception data except the to-be-associated perception target in the first target association pair as first unassociated perception data. Take the fusion target in the fusion target list except the fusion target in the first target association pair as first unassociated fusion target. Take the first target association pair, the first unassociated perception data and the first unassociated fusion target as the first association result.

4. The method of claim 1, wherein, The second association result of the first unassociated perception data and the first unassociated fusion target is determined according to a second data association strategy, and the second association result comprises the following steps. Traverse the first unassociated perception data and the first unassociated fusion target. Determine whether there is at least one frame of first unassociated perception data or the data in the first unassociated fusion target belongs to the data fusion region. If yes, determine whether the first unassociated perception data satisfies a preset check condition. If yes, perform intersection over union calculation on the different types of perception data in the first unassociated perception data and the first unassociated fusion target to determine a first association matrix. Input the first association matrix into a Hungarian algorithm to obtain a second target association pair, second unassociated perception data and second unassociated fusion target, and take the second target association pair, the second unassociated perception data and the second unassociated fusion target as the second association result.

5. The method of claim 1, wherein, The third association result of the second unassociated perception data and the second unassociated fusion target is determined according to a third data association strategy, and the third association result comprises the following steps. Traverse the second unassociated perception data and the second unassociated fusion target. Determine whether the data in the second unassociated perception data and the second unassociated fusion target all belong to other regions and the second unassociated perception data is radar perception data. If yes, determine whether the second unassociated perception data satisfies a preset check condition. If yes, perform distance calculation on the same types of perception data in the second unassociated perception data and the second unassociated fusion target to determine a second association matrix. Input the second association matrix into a Hungarian algorithm to obtain a third target association pair, third unassociated perception data and third unassociated fusion target, and take the third target association pair, the third unassociated perception data and the third unassociated fusion target as the third association result.

6. The method of claim 1, wherein, The data association result is determined according to the first association result, the second association result and the third association result. obtaining a first target association pair in the first association result, a second target association pair in the second association result, a third target association pair in the third association result, third unassociated perception data, and third unassociated fusion target; taking the first target association pair, the second target association pair, and the third target association pair as target association pairs in the data association result; taking the third unassociated perception data as unassociated perception data in the data association result; taking the third unassociated fusion target as unassociated fusion target in the data association result.

7. The method of claim 1, wherein, The updating of the fusion target list according to the data association result comprises: updating target attributes of fusion target association in the fusion target list according to target association pairs in the data association result; creating a new fusion target according to unassociated perception data in the data association result and storing the new fusion target in the fusion target list; deleting an expired fusion target in the fusion target list according to unassociated fusion target in the data association result.

8. The method of claim 7, wherein, The updating of the target attributes of fusion target association in the fusion target list according to the target association pairs in the data association result comprises: determining target attributes to be added according to to-be-fused perception data corresponding to the target association pairs; determining a fusion target corresponding to the target association pairs as a target to be added from the fusion target list; and adding the target attributes to be added to the target to be added.

9. The method of claim 8, wherein, The determining of the target attributes to be added according to the to-be-fused perception data corresponding to the target association pairs in the data association result comprises: if the to-be-fused perception data corresponding to the target association pairs is visual perception data, determining a first proportion value of first perception data in the to-be-fused perception data according to an association duration of the to-be-fused perception data; determining first target attributes according to the first proportion value and the first perception data, and taking second perception data in the to-be-fused perception data and the first target attributes as target attributes to be added; if the to-be-fused perception data corresponding to the target association pairs is radar perception target data, determining a second proportion value of third perception data in the to-be-fused perception data according to an association duration of the to-be-fused perception data; determining third target attributes according to the second proportion value and the third perception data, and taking the third target attributes as target attributes to be added.

10. The method of claim 7, wherein, The creating of the new fusion target according to the unassociated perception data in the data association result and the storing of the new fusion target in the fusion target list comprise: determining a category of the unassociated perception data; if the unassociated perception data is visual perception data in a target creation area or a data fusion area, creating a new fusion target and storing the new fusion target in the fusion target list; adding the unassociated perception data as target attributes to the new fusion target; if the unassociated perception data is radar perception target data, not updating the fusion target list.

11. The method of claim 7, wherein, The deleting of the expired fusion target in the fusion target list according to the unassociated fusion target in the data association result comprises: If an unassociated fusion target has an unassociated time greater than a set time threshold, the unassociated fusion target is determined as an expired fusion target; The expired fusion target is deleted from the fusion target list.

12. The method of claim 1, wherein, The fusion target information of the perception area is determined according to associated perception data corresponding to each fusion target in the fusion target list, including: Associated perception data of each fusion target in the fusion target list is obtained; The associated perception data is converted according to a set coordinate system to obtain fusion target information in the perception area after conversion.

13. A multi-sensor based roadside perception apparatus, characterized by, It includes: A data acquisition module is configured to acquire to-be-fused perception data of a perception area; A target association module is configured to determine a data association result of the to-be-fused perception data and a fusion target list according to the to-be-fused perception data and the pre-created fusion target list; A data fusion module is configured to update the fusion target list according to the data association result; An information determination module is configured to determine fusion target information of the perception area according to associated perception data corresponding to each fusion target in the fusion target list; The perception area is divided into a target creation area, a data fusion area and other areas in sequence according to a target moving direction, the target creation area corresponds to visual perception data, the data fusion area corresponds to visual perception data and radar perception data, and the other areas correspond to radar perception data; The target creation area is a near-distance area with camera detection accuracy higher than radar detection accuracy, and is used for identifying targets and maintaining static attribute information of fusion targets only through visual perception data captured by a camera; The data fusion area is a middle-distance area with camera and radar satisfying detection accuracy, and is used for maintaining dynamic attribute information of fusion targets through visual perception data captured by a camera and radar perception data captured by a radar; The other areas are far-distance areas with radar detection accuracy higher than camera detection accuracy, and are used for maintaining dynamic attribute information of fusion targets only through radar perception data captured by a radar; The target association module includes: A second association unit is configured to determine a data association result of to-be-fused perception data and a fusion target list according to a preset data association strategy if the fusion target list is not empty; The second association unit specifically includes: A first determination subunit is configured to determine a first association result of to-be-fused perception data and a fusion target list according to a first data association strategy; A second determination subunit is configured to determine a second association result of first unassociated perception data and first unassociated fusion targets according to a second data association strategy if the first association result contains the first unassociated perception data and the first unassociated fusion targets; A third determination subunit is configured to determine a third association result of second unassociated perception data and second unassociated fusion targets according to a third data association strategy if the second association result contains the second unassociated perception data and the second unassociated fusion targets; A fourth determination subunit is configured to determine a data association result according to the first association result, the second association result and the third association result. The first data association strategy refers to association by identification number, the second data association strategy refers to association by intersection over union, and the third data association strategy refers to association by distance.

14. An edge computing device, comprising: Comprise: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-sensor-based roadside perception method of any one of claims 1-12.

15. A storage medium containing computer-executable instructions, wherein: The computer executable instructions, when executed by a computer processor, are used to execute the multi-sensor-based roadside perception method of any one of claims 1-12.

Images