Asynchronous fusion method, device and system for road side pre-fusion multi-sensor

Abstract

The invention provides an asynchronous fusion method, device and system for road side pre-fusion multiple sensors. The method comprises the following steps: performing splicing and pre-fusion on original data which is collected by the multiple sensors at the road side and aims at a specified target at a data level to form pre-fusion data; and carrying out feature vector extraction transformation and association on the pre-fused data, and carrying out asynchronous fusion processing without clock synchronization dependence. The method is suitable for various fields of traffic perception.

Description

Technical Field

[0001] The present application relates to the field of roadside perception technology, and more specifically, to an asynchronous fusion method, device and system for roadside front fusion multi-sensor. Background Art

[0002] In the continuous construction of smart transportation, roadside perception generally uses camera sensors combined with roadside edge computing to achieve intelligent perception of traffic participants, road conditions, etc. However, the existing roadside perception technologies and methods have many significant defects and limitations.

[0003] In actual applications, most roadside perception systems simply use cameras as the main data source. However, cameras cannot work normally under severe weather conditions (such as heavy fog, heavy rain, sandstorms, etc.) and insufficient light, rainy and foggy weather scenes, and the perception effect will be greatly reduced or even completely fail. If millimeter-wave radar sensors are used, they cannot be used in closed or semi-enclosed spaces such as indoor spaces and tunnels.

[0004] Secondly, existing roadside perception methods generally use post-fusion to process multi-sensor data. Although this method can integrate information from multiple data sources, it will lose the initial information of the original data, thus affecting the accuracy and effect of perception.

[0005] Furthermore, centralized computing also has extremely high requirements for data synchronization. Inconsistent timestamps between multiple sensors will increase the complexity and delay of data processing. Such a system with extremely strong clock synchronization requirements will not work properly if the clock source is unstable or lost. Summary of the invention

[0006] In view of this, the present application is proposed in consideration of the above problems.

[0007] The first purpose of the present application is to design an asynchronous fusion method for roadside front-fusion multi-sensors, which requires splicing and fusion of the original data first, and further feature extraction, transformation and association of the data after the front fusion, thereby overcoming the defect of the post-fusion technology in the prior art that the initial information of the original data is lost, resulting in poor perception effect; further, the front-fusion data needs to be asynchronously fused without clock synchronization dependence, thereby overcoming the strict dependence of the centralized computing unit on high clock synchronization in the prior art, and the problem of overall perception performance degradation caused by data synchronization delay.

[0008] The second objective of the present application is to design a device for implementing an asynchronous fusion method of roadside front fusion multi-sensor.

[0009] The third objective of the present application is to design a system for implementing an asynchronous fusion method of roadside front fusion multi-sensor.

[0010] In order to achieve the above technical objectives, this application is proposed: The present application provides an asynchronous fusion method for roadside front fusion multi-sensor, comprising the following steps: At the data level, the raw data collected by multiple sensors on the roadside for the designated target are spliced ​​and pre-fused to form pre-fused data, and feature vectors are extracted, transformed, and associated on the pre-fused data; Asynchronous fusion processing without clock synchronization dependency is performed on the pre-fusion data.

[0011] Exemplarily, the following operations are further performed before the splicing and pre-fusion to form pre-fusion data: The raw data of the designated target is screened by a preset program to remove background noise in the raw data of the designated target.

[0012] Exemplarily, the extraction, transformation and association steps of the pre-fusion data are: The pre-fusion data is subjected to multiple convolution processes to obtain the pre-fusion data features, and the feature vector representing the pre-fusion data is extracted; According to the time sequence of the actual arrival of the front fusion data transmitted by each sensor, the features of the front fusion data are orderly associated with each other using the ciou algorithm; The ciou algorithm is:

[0013] ; In the IOU formula, A is the detection box and B is the true target box; in the CIOU formula, IOU is the union intersection ratio of the detection box and the true target box, ρ2(b,bgt) represents the Euclidean distance between the two center points of the detection box and the true target box, where b represents the detection box and bgt represents the true target box; c represents the diagonal distance of the minimum closed convex surface; α is a parameter used to balance the proportion; v is a parameter to measure whether the proportion between the detection box and the true target box is consistent.

[0014] Exemplarily, before performing asynchronous fusion processing without clock synchronization dependency, a method for processing the feature vector of the pre-fusion data includes: Comparing the feature vector of the pre-fusion data with the stored historical track data to determine whether the pre-fusion data belongs to the existing historical track data; If the historical track data of the pre-fusion data fails to match, the stored historical track data is used for prediction, and the position where it may appear in the future frame is predicted; If the historical track data of the previous fusion data is matched successfully, it means that the current previous fusion data is associated with one of the stored historical track data, and the previous fusion data is iteratively filtered to obtain the target track of the designated target with high accuracy.

[0015] Exemplarily, the iterative filtering processing step includes: The pre-fusion data successfully matched to the historical track is used as input data, and an iterative filtering process is performed, and the burrs in the pre-fusion data are removed by using the stored historical track data.

[0016] Exemplarily, a variety of sensors are deployed in the same perception area, and the diverse sensors include cameras, lidars, and millimeter-wave radars.

[0017] The present application also provides a device for realizing an asynchronous fusion method of roadside front fusion multi-sensor, comprising: Pre-fusion equipment, primary fusion equipment; The pre-fusion device splices and pre-fuses the original data collected by multiple sensors for the specified target, extracts, transforms and associates the feature vectors of the formed pre-fusion data, and then uses the main fusion device to perform asynchronous fusion processing on the pre-fusion data without clock synchronization dependence.

[0018] Exemplarily, the device further comprises: Switches, backup fusion equipment, cloud servers, 5G base stations, mobile smart terminals, and vehicle-mounted smart cockpits; The switch is used to output the pre-fusion data / data after asynchronous fusion processing without clock synchronization dependence; the backup fusion device takes over the work when the main fusion device fails to work; the cloud server stores and processes the data after asynchronous fusion processing without clock synchronization dependence on the main fusion device, and transmits the stored and processed data after asynchronous fusion processing without clock synchronization dependence to the mobile phone smart terminal and the vehicle-mounted smart cockpit through the 5G base station.

[0019] The present application also provides a system for realizing an asynchronous fusion method of roadside front fusion multi-sensor, comprising: Pre-fusion module: used to splice and pre-fuse the original data of the designated target collected by multiple sensors on the roadside to form pre-fusion data at the data level, and extract, transform and associate the feature vectors of the pre-fusion data; Asynchronous fusion module: used for performing asynchronous fusion processing on the pre-fusion data without clock synchronization dependency.

[0020] Exemplarily, the front fusion module contains: Preprocessing module: before splicing and pre-fusion to form pre-fusion data, the original data of the designated target is screened through a preset program to remove background noise in the original data of the designated target; Feature extraction module and association module: used to obtain the features of the pre-fusion data after multiple convolution processes on the pre-fusion data, and extract the feature vector representing the pre-fusion data, and use the ciou algorithm to perform orderly association processing on the pre-fusion data features according to the actual arrival time sequence of the pre-fusion data transmitted by each sensor.

[0021] It should be understood that the above description is an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps. Figure 1 A schematic flow chart of an asynchronous fusion method for roadside front fusion multi-sensor provided in an embodiment of the present application; Figure 2 A flowchart of the final fused track outputted by a designated target in an asynchronous fusion method of a roadside front fusion multi-sensor provided in an embodiment of the present application; Figure 3 A diagram of an asynchronous fusion device of a roadside front fusion multi-sensor provided in an embodiment of the present application, wherein 101-front fusion device, 102-switch, 103-cloud server, 104-vehicle intelligent cockpit, 105-5G base station, 106-mobile smart terminal, 107-main fusion device, 108-backup fusion device.

[0023] Specific implementation method In order to make the purpose, technical solutions and advantages of the present application more obvious, the example embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application described in the present application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "original", "target", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0025] This application provides an asynchronous fusion method for roadside front fusion multi-sensor, referring to Figure 1 , which shows a schematic flow chart of an asynchronous fusion method for roadside front fusion multi-sensor provided in an embodiment of the present application. The method includes steps S110-S120: Step S110: At the data level, the original data collected by multiple sensors on the roadside for the designated target are spliced ​​and pre-fused to form pre-fused data, and feature vectors are extracted, transformed, and associated on the pre-fused data.

[0026] The step S110 is performed in the pre-fusion module, in which we first perform splicing and pre-fusion processing on the data level for the raw data collected by various sensors deployed on the roadside for the designated target. This processing process aims to utilize the complementarity of multiple sensors to improve the accuracy and reliability of the data.

[0027] Preferably, multiple sensors of different types are arranged in the same perception area, such as lidar, millimeter-wave radar, high-definition camera, and infrared sensor. The collaborative work of multiple sensors can not only avoid the limitations of a single type of multi-sensor in complex environments (such as bad weather, tunnels, night, etc.), for example, detecting the same designated target as multiple designated targets, and causing missed detection and false detection of designated targets; it can also achieve the complementarity of raw data, thereby improving the accuracy and reliability of perception, and ensuring the all-weather, all-scenario, and high-precision collection of raw data for designated targets.

[0028] This technical solution does not limit the perceived targets, including but not limited to motor vehicles, non-motor vehicles, buildings, falling rocks, ground collapse, foreign objects, etc., and the identification of various special objects.

[0029] Those skilled in the art can select the desired type according to actual specific needs, and this application does not impose any limitation.

[0030] Specifically, the step S110 includes the following steps: before collecting the raw data for the specified target, the raw data must be preprocessed. The specific preprocessing method is: Screening of raw data in the preprocessing module: Screening the raw data of the designated target through a preset program to remove background noise in the raw data of the designated target.

[0031] Preferably, in this embodiment, the screening of the designated target raw data utilizes a preset DP iterative filter program to screen the raw data, remove background noise, and improve the quality of the raw data.

[0032] At the data level, the raw data collected by multiple sensors on the roadside for the designated target are spliced ​​and pre-fused to form pre-fused data. The steps of splicing and pre-fusion of the raw data are as follows: The filtered raw data is spliced ​​and processed in the pre-fusion device to form pre-fusion data. This step makes full use of the complementarity of different sensors in data form and perception principle, laying a solid foundation for subsequent feature extraction and association.

[0033] Specifically, in this embodiment, the raw data output by the radar and the camera are significantly different. The radar perceives the environment by emitting and receiving electromagnetic waves, and the raw data it outputs are mainly laser point cloud voxels and millimeter-wave thermal maps. The millimeter-wave thermal map is a four-dimensional image obtained by Fourier transform. The camera forms an image by capturing changes in light, and the raw data it outputs is a two-dimensional or three-dimensional image sequence. Due to the differences in perception principles and data forms between radar and cameras, the collected raw data of the designated target need to be spliced ​​and fused.

[0034] Furthermore, the above-mentioned fusion of the present application uses a pre-fusion technology, which is to directly splice and pre-fuse the raw data from different sensors at the data level through the pre-fusion device 101 end; using the raw data without human intervention, it can be ensured to the greatest extent that the original features of these raw data are completely preserved. These raw data play an important role in subsequent processing and recognition. Through the direct fusion of the raw data by the pre-fusion technology, and utilizing the complementarity between the raw data in the pre-fusion process, the output pre-fusion data can provide more comprehensive and richer information, which helps to improve the accuracy of identifying the designated target; and compared with the post-fusion technology in the existing roadside perception method, the pre-fusion technology performs pre-fusion in the raw data collection stage. The pre-fusion method does not need to wait for all sensor data to be fully synchronized before processing. It allows the sensor raw data to arrive asynchronously and be fused in real time, reducing the delay in the raw data synchronization process, so that the output pre-fusion data has lower delay and faster response speed, thereby improving the real-time performance of the system of the present application.

[0035] The front fusion technology requires certain computing resources at the front fusion device 101, but compared with the traditional centralized computing method, the front fusion technology in this application can reduce the demand for synchronization and processing of raw data, thereby reducing the dependence on high-computing servers, which is conducive to reducing the overall cost of this application. At the same time, because the front fusion technology can more accurately identify the specified target, it can reduce the situation of false alarms and missed alarms.

[0036] The steps of extracting, transforming and associating the feature vector of the pre-fusion data in step S110 are: The pre-fusion data is subjected to multiple convolution processes to obtain the pre-fusion data features, and the feature vector representing the pre-fusion data is extracted; According to the time sequence of the actual arrival of the front fusion data transmitted by each sensor, the features of the front fusion data are orderly associated using the ciou algorithm. This step is performed in the feature extraction module and the association module.

[0037] Specifically, the ciou algorithm is:

[0038] ; Among them, IOU is the detection box and B is the real target box; CIOU is the union intersection ratio of the detection box and the real target box, ρ2(b,bgt) represents the Euclidean distance between the two center points of the detection box and the real target box, where b represents the detection box and bgt represents the real target box; c represents the diagonal distance of the minimum closed convex surface; α is a parameter used to balance the proportion; v is a parameter to measure whether the proportion between the detection box and the real target box is consistent.

[0039] The CIOU algorithm achieves high-precision feature association by comprehensively considering factors such as the intersection-in-union ratio, center point distance, diagonal distance, and proportion consistency between the detection box and the true target box.

[0040] In the embodiment, the pre-fusion data formed in the pre-fusion device 101 in step S110 is further transmitted to the main fusion device 107 through the data transmission channel of the switch 102, and step S120 is performed: asynchronous fusion processing without clock synchronization dependency is performed on the pre-fusion data.

[0041] Specifically, S120 includes the following steps: before the pre-fusion data is subjected to asynchronous fusion processing without clock synchronization dependency, the pre-fusion data needs to be matched with historical tracks. Specifically, the steps of historical track matching are: The feature vector of the pre-fusion data is compared with the stored historical track data to determine whether the pre-fusion data belongs to the existing historical track data.

[0042] If the historical track data of the pre-fusion data fails to match, the stored historical track data is used for prediction, and the position where it may appear in the future frame is predicted; If the historical track data of the previous fusion data is matched successfully, it means that the current previous fusion data is associated with one of the stored historical track data, and the previous fusion data is iteratively filtered to obtain the target track of the designated target with high accuracy.

[0043] Furthermore, the iterative filtering process comprises: using the pre-fusion data successfully matched to the historical track as input data, performing iterative filtering, and using the stored historical track data to remove burrs in the pre-fusion data to generate a smoother and more accurate target track.

[0044] In step S120, after the pre-fusion data is subjected to asynchronous fusion processing without clock synchronization dependency, it needs to be output. The specific output steps are: After the above processing, asynchronous fusion data without clock synchronization dependence is obtained. These asynchronous fusion data contain consistent explanations and descriptions of the specified target, such as precise position, speed, heading angle and other key information.

[0045] Specifically, the above method outputs the final fused track reference for the specified target. Figure 2 shown.

[0046] Specifically, this method avoids the need for traditional clock synchronization, asynchronously processes the non-synchronous time of the pre-fusion data of the specified target, and cleverly fuses the raw data from different sensors about the specified target through the asynchronous fusion algorithm, ultimately generating a consistent interpretation and description of the specified target.

[0047] The use of the asynchronous fusion method completely gets rid of the constraints of time synchronization and seamlessly processes the asynchronously outputted pre-fusion data. The asynchronous fusion method processes the diversified raw data from multiple sensors directly on the pre-fusion device 101 through the pre-fusion technology. Such processing further ensures the continuity and real-time processing of the raw data and effectively reduces the overall perception delay. Even if a pre-fusion device 101 is disconnected or fails, the raw data on the sensor can be processed asynchronously without affecting the subsequent output of the entire system. The asynchronous fusion method does not require a clock source for clock synchronization, so that each pre-fusion device 101 can work independently, reducing the dependence on the synchronization mechanism. Furthermore, through the application of pre-fusion technology and deep learning algorithms, the asynchronous fusion algorithm can retain the initial information and complementarity of the original data to the maximum extent during the fusion process, thereby improving the accuracy and reliability of target recognition.

[0048] In addition, the asynchronous fusion method perfectly fits the data scenarios of distributed computing systems and multiple sensors with different frequencies. Faced with the problem that the raw data output by distributed computing sensors is affected by inconsistent sensor frequencies and fluctuations in computing time, this solution abandons the complex synchronization strategy and adopts an efficient "instant fusion" mode (that is, asynchronous fusion fuses one frame of data at a time), ignoring the timestamp difference and achieving an extremely low frame loss rate, providing strong support for multi-source data fusion in a distributed environment.

[0049] In order to implement the above method, the present application also discloses an asynchronous fusion device for roadside front fusion multi-sensor, referring to Figure 3 As shown, the device comprises: Pre-fusion device 101, main fusion device 107, switch 102, backup fusion device 108, cloud server 103, 5G base station 105, mobile smart terminal 106, vehicle-mounted smart cockpit 104; As the first checkpoint of data processing, the pre-fusion device 101 is responsible for receiving raw data from various sensors (including lidar, camera, millimeter-wave radar, etc.), and performing efficient splicing and pre-fusion processing for designated targets. This process not only integrates the complementary advantages of various types of data, but also quickly completes the initial feature vector extraction and transformation through the built-in low-computing power computing unit, laying a solid foundation for reducing the pressure of pre-fusion data transmission and reducing transmission delay for subsequent processing.

[0050] As the main equipment of the device, the main fusion device 107 demonstrates its excellent processing ability without clock synchronization. The asynchronous fusion algorithm is used to integrate the asynchronous pre-fusion data from each pre-fusion device 101 without time synchronization restrictions to form asynchronous fusion data, thereby realizing the comprehensive analysis and construction of key information such as the precise position, speed, and heading angle of the target object. This process completely gets rid of the dependence of the traditional centralized system on time synchronization, simplifies the system structure of this application, and improves the overall efficiency.

[0051] As a high-speed channel for data transmission, the switch 102, with its high bandwidth and low latency characteristics, ensures that the pre-fusion data can flow seamlessly and quickly to the main fusion device 107 and the cloud server 103. The efficient operation of this link provides a strong guarantee for the real-time response and data accuracy of the system of this application.

[0052] To ensure the continuous and stable operation of the device, the backup fusion device 108 automatically takes over the work when the main fusion device 107 fails, which effectively avoids service interruption and enhances the reliability of the system of this application.

[0053] The cloud server 103 is responsible for storing and deeply processing the asynchronous fusion data output by the main fusion device 107, and pushes the processed road condition information to the mobile smart terminal 106 and the vehicle-mounted smart cockpit 104 in real time through the high-speed network connection of the 5G base station 105. This step not only improves the efficiency of information transmission, but also enables users to enjoy a more intelligent and personalized service experience.

[0054] The mobile phone smart terminal 106 receives real-time traffic information to provide users with accurate walking navigation, violation warning and other intimate services; the in-vehicle smart cockpit 104 combines the vehicle's own sensor data to realize advanced functions such as vehicle warning and map twin, creating a safe and intelligent driving environment for drivers. This series of applications not only improves the level of intelligent road traffic, but also brings great convenience to users' travel.

[0055] To sum up, this application proposes a device for realizing the asynchronous fusion method of roadside front fusion multi-sensor. Through the coordinated use of the above-mentioned devices, an efficient, reliable and low-cost roadside perception device suitable for smart city construction is formed. The collaborative work between the various devices ensures the real-time collection, processing and transmission of data, and provides strong support for multiple application scenarios such as urban traffic management, vehicle-road collaboration, and violation supervision.

[0056] Based on the same inventive concept, an embodiment of the present application proposes an asynchronous fusion system of roadside front fusion multi-sensor, which realizes efficient data processing through a front fusion module and an asynchronous fusion module.

[0057] The pre-fusion module first integrates the raw data from multiple sensors, splices and pre-fuses them for the specified target, and forms pre-fusion data. In this process, the pre-processing module is responsible for screening and removing background noise in the raw data to ensure data quality. Subsequently, the feature extraction module works in conjunction with the association module to extract the feature vector of the pre-fusion data through convolution processing, and uses the CIOU algorithm to perform orderly association according to the order in which the sensor data actually arrives, so as to preserve the temporal order and association of the data.

[0058] The asynchronous fusion module further processes these pre-fusion data to achieve asynchronous fusion without clock synchronization dependence, improving the flexibility and reliability of the system. The entire system aims to provide accurate and real-time data support for application scenarios such as smart city traffic management and vehicle-road collaboration through the fusion processing of multi-sensor data.

[0059] It can be understood that the above description of the asynchronous fusion device and system for roadside front fusion multi-sensor can implement the steps of the asynchronous fusion method for roadside front fusion multi-sensor provided in the aforementioned embodiment, which will not be repeated here.

[0060] In summary, the asynchronous fusion method of a front-fusion multi-sensor proposed in this application realizes the perception capability of high precision, low latency, high reliability, low cost, and full-scenario adaptability by integrating the asynchronous fusion device and system of the front-fusion multi-sensor; this application can not only simultaneously identify multiple traffic targets and special objects, output precise coordinates and heading angles, but also work around the clock and adapt to harsh environments. Its advantage is that the disconnection of any device does not affect the overall output, and low latency and high-precision positioning can be guaranteed without clock synchronization, covering a wide range of smart city fields. In addition, the asynchronous parallel computing architecture is adopted, which is lower in cost and stable, and supports large-scale deployment. The front-fusion technology deeply integrates multi-source data, further improves the perception accuracy and efficiency, and provides strong support for vehicle-road collaboration, traffic management, etc. The technology of this application not only simplifies the complexity of the system, but also reduces the overall cost, injecting new impetus into the transportation of the intelligent field.

[0061] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 Method in the embodiment shown.

[0062] According to another aspect of the embodiment of the present application, an electronic device is provided, the electronic device comprising: a processor and a memory. The memory stores a computer program, and the processor is used to execute the computer program to implement the above method.

[0063] A person skilled in the art can easily understand the implementation structure, working principle and beneficial effects of the electronic device and the computer-readable storage medium by reading the above method, and will not be described in detail for the sake of brevity.

[0064] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0065] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0066] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0067] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0068] It should be noted that the above embodiments illustrate rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation to the claims.

[0069] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A roadside front fusion multi-sensor asynchronous fusion method, characterized in that: The following steps are involved: At the data level, the raw data collected by multiple sensors on the roadside for the designated target are spliced ​​and pre-fused to form pre-fused data, and feature vectors are extracted, transformed, and associated on the pre-fused data; Asynchronous fusion processing without clock synchronization dependency is performed on the pre-fusion data.

2. The method according to claim 1, characterized in that Before the splicing and pre-fusion form the pre-fusion data, the following operations are also performed: The raw data of the designated target is screened by a preset program to remove background noise in the raw data of the designated target.

3. The method according to claim 1, characterized in that The steps of extracting, transforming and associating the pre-fusion data are as follows: The pre-fusion data is subjected to multiple convolution processes to obtain the pre-fusion data features, and the feature vector representing the pre-fusion data is extracted; According to the time sequence of the actual arrival of the front fusion data transmitted by each sensor, the features of the front fusion data are orderly associated with each other using the ciou algorithm; The ciou algorithm is: ； In the IOU formula, A is the detection box and B is the real target box; In the CIOU formula, IOU is the intersection-over-union ratio of the detection box and the true target box, ρ2(b,bgt) represents the Euclidean distance between the two center points of the detection box and the true target box, where b represents the detection box and bgt represents the true target box; c represents the diagonal distance of the minimum closed convex surface; α is a parameter used to balance the proportion; v is a parameter to measure whether the proportion between the detection box and the true target box is consistent.

4. The method according to claim 1, characterized in that: Before performing asynchronous fusion processing without clock synchronization dependency, a method for processing the feature vector of the pre-fusion data includes: Comparing the feature vector of the pre-fusion data with the stored historical track data to determine whether the pre-fusion data belongs to the existing historical track data; If the historical track data of the pre-fusion data fails to match, the stored historical track data is used for prediction, and the position where it may appear in the future frame is predicted; If the historical track data of the previous fusion data is matched successfully, it means that the current previous fusion data is associated with one of the stored historical track data, and the previous fusion data is iteratively filtered to obtain the target track of the designated target with high accuracy.

5. The method according to claim 4, characterized in that: The iterative filtering process step comprises: The pre-fusion data successfully matched to the historical track is used as input data, and an iterative filtering process is performed, and the burrs in the pre-fusion data are removed by using the stored historical track data.

6. The method according to claim 1, characterized in that A variety of sensors are deployed in the same perception area, including cameras, lidars and millimeter-wave radars.

7. A device for implementing the method according to any one of claims 1 to 6, characterized in that: include: Pre-fusion equipment, primary fusion equipment; The pre-fusion device splices and pre-fuses the original data collected by multiple sensors for the specified target, extracts, transforms and associates the feature vectors of the formed pre-fusion data, and then uses the main fusion device to perform asynchronous fusion processing on the pre-fusion data without clock synchronization dependence.

8. The device according to claim 7, characterized in that The device also includes: Switches, backup fusion equipment, cloud servers, 5G base stations, mobile smart terminals, and vehicle-mounted smart cockpits; The switch is used to output the pre-fusion data / data after asynchronous fusion processing without clock synchronization dependence; the backup fusion device takes over the work when the main fusion device fails to work; the cloud server stores and processes the data after asynchronous fusion processing without clock synchronization dependence on the main fusion device, and transmits the stored and processed data after asynchronous fusion processing without clock synchronization dependence to the mobile phone smart terminal and the vehicle-mounted smart cockpit through the 5G base station.

9. A system for implementing the method according to any one of claims 1 to 6, characterized in that: include: Pre-fusion module: used to splice and pre-fuse the original data of the designated target collected by multiple sensors on the roadside to form pre-fusion data at the data level, and extract, transform and associate the feature vectors of the pre-fusion data; Asynchronous fusion module: used for performing asynchronous fusion processing on the pre-fusion data without clock synchronization dependency.

10. The system according to claim 9, characterized in that The front fusion module contains: Preprocessing module: before splicing and pre-fusion to form pre-fusion data, the original data of the designated target is screened through a preset program to remove background noise in the original data of the designated target; Feature extraction module and association module: used to obtain the features of the pre-fusion data after multiple convolution processes on the pre-fusion data, and extract the feature vector representing the pre-fusion data, and use the ciou algorithm to perform orderly association processing on the pre-fusion data features according to the actual arrival time sequence of the pre-fusion data transmitted by each sensor.

Images