Method and apparatus for multi-sensor signal fusion, electronic device, and storage medium

Abstract

The application discloses a kind of multi-sensor signal fusion method, device, electronic equipment and storage medium, belong to intelligent monitoring technical field.It includes: obtaining the sampling signal of each detection sensor in multiple detection sensors at the same time, for the same detection range, the working frequency band of multiple detection sensors is different.Each sensor sampling signal in multiple sensor sampling signals is synchronized to the same feature dimension space, to obtain multiple signal data, wherein the feature dimension space is set according to the detection dimension of each detection sensor and system output requirement.Detecting multiple signal data fusion processing, to obtain spatial signal description matrix, spatial signal description matrix can describe the signal echo quality of target.Based on spatial signal description matrix, output fusion data.In this way, based on the original sampling signal of detection sensor data fusion, avoid fusion to appear missing detection etc.Case, improve the accuracy of the final generated fusion data.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring technology, and in particular to a method, apparatus, electronic device and storage medium for multi-sensor signal fusion. Background Technology

[0002] Detection sensors are widely used in the field of intelligent monitoring technology. Common detection sensors include, but are not limited to, lidar sensors, millimeter-wave radar sensors, and visible light sensors. Different detection sensors operate in different frequency bands, resulting in differences in their detection performance. For example, millimeter-wave radar sensors have stronger penetration capabilities than lidar sensors, but lidar sensors have higher detection accuracy than millimeter-wave radar sensors.

[0003] Because the detection performance of a single sensor has certain limitations, multiple sensor fusion is usually employed to improve detection robustness. Most current fusion methods are based on point cloud data or target detection results, which are obtained after a series of filtering and processing steps.

[0004] However, if a single detection sensor encounters challenges during detection, such as a lidar sensor encountering severe weather conditions like smog or sandstorms, it may cause some blurry but real target-related data to be filtered out during the screening and other processing. This can lead to missed detections during fusion, reducing the accuracy of the final fused data. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for multi-sensor signal fusion, which can solve the problem in related technologies where missed detections occur during fusion, leading to reduced accuracy of the final generated fused data. The technical solution is as follows:

[0006] Firstly, a method for multi-sensor signal fusion is provided, the method comprising:

[0007] The sampling signals of each of the multiple detection sensors targeting the same detection range at the same time are acquired to obtain multiple sensor sampling signals, wherein the multiple detection sensors operate in different frequency bands;

[0008] Each sensor sampling signal from the plurality of sensor sampling signals is synchronized to the same feature dimension space to obtain multiple signal data, wherein the feature dimension space is set according to the detection dimension of each detection sensor and the system output requirements;

[0009] The multiple signal data are fused to obtain a spatial signal description matrix, which can describe the signal echo quality of the target.

[0010] Based on the spatial signal description matrix, fused data is output.

[0011] As an example of this application, the fusion processing of the multiple signal data includes:

[0012] Based on each of the plurality of signal data, at least one evaluation parameter capable of evaluating the signal echo quality of the target is determined, resulting in a plurality of evaluation parameter sets, each of the plurality of evaluation parameter sets including the at least one evaluation parameter;

[0013] Signal fusion processing is performed based on the aforementioned set of multiple evaluation parameters.

[0014] As an example of this application, the plurality of signal data includes first laser signal data and millimeter-wave signal data;

[0015] The step involves determining at least one evaluation parameter capable of assessing the signal echo quality of the target based on each of the plurality of signal data, resulting in a set of multiple evaluation parameters, including:

[0016] Based on the millimeter-wave signal data, the position of the millimeter-wave peak and the millimeter-wave echo energy corresponding to the target are determined, and a set of evaluation parameters is obtained.

[0017] Based on the first laser signal data, the echo leading edge position and laser peak saturation corresponding to the target are determined to obtain another set of evaluation parameters. The echo leading edge position refers to the position detected by the lidar sensor when the target is first detected by the lidar sensor, and the laser peak saturation is used to indicate the confidence level that the target is a real target.

[0018] As an example of this application, the signal fusion processing based on the multiple sets of evaluation parameters includes:

[0019] Based on the millimeter wave peak position and the millimeter wave echo energy, the variance of the millimeter wave peak position is determined to obtain a first variance, which is used to indicate the probability distribution of the target at the millimeter wave peak position.

[0020] Based on the echo leading edge position and the laser peak saturation, the variance of the echo leading edge position is determined to obtain a second variance, which is used to indicate the probability distribution of the target at the echo leading edge position.

[0021] The spatial signal description matrix is ​​determined based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance.

[0022] As an example of this application, determining the spatial signal description matrix based on the millimeter-wave peak position, the echo leading edge position, the first variance, and the second variance includes:

[0023] Based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance, the spatial signal description matrix is ​​determined using the following formulas (1) to (3):

[0024] (1)

[0025] (2)

[0026] (3)

[0027] Among them, the include( ), the The millimeter wave peak position includes ( ), the The position of the leading edge of the echo includes ( ), the For the first variance, the For the second variance, the ( ) is a variable, the V represents the joint confidence level, and V is the joint confidence level distribution.

[0028] As an example of this application, the step of outputting fused data based on the spatial signal description matrix includes:

[0029] From the spatial signal description matrix, determine the joint positions where the joint confidence level is greater than the confidence threshold;

[0030] The joint positions with a joint confidence level greater than a confidence threshold and their corresponding joint confidence levels are output as the fused data.

[0031] As an example of this application, the fusion processing of the multiple signal data includes:

[0032] Location association is performed based on the multiple signal data;

[0033] Based on the location association results, the signal data of the target is determined from each of the multiple signal data, thereby obtaining multiple target signal data of the target;

[0034] The multiple target signal data are fused together.

[0035] As an example of this application, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor, and the plurality of signal data includes first laser signal data and millimeter-wave signal data;

[0036] Before performing location association based on the multiple signal data, the method further includes:

[0037] First dynamic target signal data is obtained by filtering out the signal data of static targets detected by the lidar sensor from the first laser signal data, and second dynamic target signal data is obtained by filtering out the signal data of static targets detected by the millimeter-wave radar sensor from the millimeter-wave signal data.

[0038] The location association based on the multiple signal data includes:

[0039] Position association is performed based on the first dynamic target signal data and the second dynamic target signal data.

[0040] As an example of this application, the millimeter-wave signal data includes velocity, and the step of filtering out the signal data of the static target detected by the millimeter-wave radar sensor from the millimeter-wave signal data to obtain the second dynamic target signal data includes:

[0041] From the millimeter-wave signal data, determine the signal data whose velocity is less than the velocity threshold;

[0042] Delete the signal data whose velocity is less than the velocity threshold from the millimeter wave signal data;

[0043] The millimeter-wave signal data obtained after deletion processing is determined as the second dynamic target signal data.

[0044] As an example of this application, the step of filtering out the signal data of the static target detected by the lidar sensor from the first laser signal data to obtain the first dynamic target signal data includes:

[0045] Acquire second laser signal data, which is a frame of laser signal data adjacent to the first laser signal data;

[0046] The difference between the first laser signal data and the second laser signal data is used as the second dynamic target signal data.

[0047] As an example of this application, the feature dimension space includes angular features and distance features, and the output of fused data based on the spatial signal description matrix includes:

[0048] Obtain the angle of the target in the first dynamic target signal data, and obtain the distance of the target in the second dynamic target signal data;

[0049] The angle and the distance are determined as the fused data.

[0050] As an example of this application, the method further includes:

[0051] Output the distance and angle of the static target in the first laser signal data.

[0052] As an example of this application, before synchronizing each of the plurality of sensor sampling signals to the same feature dimension space, the method further includes:

[0053] Each of the multiple sensor sampling signals is subjected to regularization processing.

[0054] The step of synchronizing each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space includes: synchronizing each sensor sampling signal after regularization processing to the feature dimension space.

[0055] As an example of this application, the method further includes:

[0056] The multiple detection sensors are synchronized in time and in space. The time synchronization refers to triggering the operation of other detection sensors in the multiple sensors by one of the multiple sensors. The spatial synchronization refers to determining the spatial transformation matrix by multiple detections of the same test target by the multiple detection sensors.

[0057] As an example of this application, the fusion processing of the multiple signal data includes:

[0058] When the sampling step sizes of the multiple detection sensors are not the same, the sampling step sizes of the multiple signal data are aligned by interpolation.

[0059] The multiple signal data that have undergone alignment processing are then fused.

[0060] As an example of this application, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor;

[0061] The step of synchronizing each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space includes:

[0062] The sampling signal of the lidar sensor is converted from the first feature dimension space to the feature dimension space. The first feature dimension space includes the feature dimensions of frame rate feature, scan beam feature, and time feature. The feature dimension space includes the feature dimensions of velocity feature, angle feature, and distance feature.

[0063] The sampling signal of the millimeter-wave radar sensor is converted from the second feature dimension space to the feature dimension space, wherein the second feature dimension space includes chirp features, radiating antenna features, and time features.

[0064] Secondly, a multi-sensor signal fusion apparatus is provided, the apparatus comprising:

[0065] The acquisition module is used to acquire the sampling signal of each of the multiple detection sensors targeting the same detection range at the same time, thereby obtaining multiple sensor sampling signals, wherein the multiple detection sensors operate in different frequency bands;

[0066] The synchronization module is used to synchronize each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space to obtain multiple signal data, wherein the feature dimension space is set according to the detection dimension of each detection sensor and the system output requirements;

[0067] The fusion module is used to fuse the multiple signal data to obtain a spatial signal description matrix, which can describe the signal echo quality of the target.

[0068] The output module is used to output fused data based on the spatial signal description matrix.

[0069] As an example of this application, the fusion module is used for:

[0070] Based on each of the plurality of signal data, at least one evaluation parameter capable of evaluating the signal echo quality of the target is determined, resulting in a plurality of evaluation parameter sets, each of the plurality of evaluation parameter sets including the at least one evaluation parameter;

[0071] Signal fusion processing is performed based on the aforementioned set of multiple evaluation parameters.

[0072] As an example of this application, the plurality of signal data includes first laser signal data and millimeter-wave signal data; the fusion module is used for:

[0073] The step involves determining at least one evaluation parameter capable of assessing the signal echo quality of the target based on each of the plurality of signal data, resulting in a set of multiple evaluation parameters, including:

[0074] Based on the millimeter-wave signal data, the position of the millimeter-wave peak and the millimeter-wave echo energy corresponding to the target are determined, and a set of evaluation parameters is obtained.

[0075] Based on the first laser signal data, the echo leading edge position and laser peak saturation corresponding to the target are determined to obtain another set of evaluation parameters. The echo leading edge position refers to the position detected by the lidar sensor when the target is first detected by the lidar sensor, and the laser peak saturation is used to indicate the confidence level that the target is a real target.

[0076] As an example of this application, the fusion module is used for:

[0077] Based on the millimeter wave peak position and the millimeter wave echo energy, the variance of the millimeter wave peak position is determined to obtain a first variance, which is used to indicate the probability distribution of the target at the millimeter wave peak position.

[0078] Based on the echo leading edge position and the laser peak saturation, the variance of the echo leading edge position is determined to obtain a second variance, which is used to indicate the probability distribution of the target at the echo leading edge position.

[0079] The spatial signal description matrix is ​​determined based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance.

[0080] As an example of this application, the fusion module is used for:

[0081] Based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance, the spatial signal description matrix is ​​determined using the following formulas (1) to (3):

[0082] (1)

[0083] (2)

[0084] (3)

[0085] Among them, the include( ), the The millimeter wave peak position includes ( ), the The position of the leading edge of the echo includes ( ), the For the first variance, the For the second variance, the ( ) is a variable, the V represents the joint confidence level, and V is the joint confidence level distribution.

[0086] As an example of this application, the output module is used for:

[0087] From the spatial signal description matrix, determine the joint positions where the joint confidence level is greater than the confidence threshold;

[0088] The joint positions with a joint confidence level greater than a confidence threshold and their corresponding joint confidence levels are output as the fused data.

[0089] As an example of this application, the fusion module is used for:

[0090] Location association is performed based on the multiple signal data;

[0091] Based on the location association results, the signal data of the target is determined from each of the multiple signal data, thereby obtaining multiple target signal data of the target;

[0092] The multiple target signal data are fused together.

[0093] As an example of this application, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor, and the plurality of signal data includes first laser signal data and millimeter-wave signal data;

[0094] The fusion module is used to: filter out the signal data of the static target detected by the lidar sensor from the first laser signal data to obtain the first dynamic target signal data, and filter out the signal data of the static target detected by the millimeter-wave radar sensor from the millimeter-wave signal data to obtain the second dynamic target signal data;

[0095] The specific implementation of the fusion module in performing position association based on the multiple signal data includes: performing position association based on the first dynamic target signal data and the second dynamic target signal data.

[0096] As an example of this application, the millimeter-wave signal data includes velocity, and the fusion module is used for:

[0097] Delete the signal data whose velocity is less than the velocity threshold from the millimeter wave signal data;

[0098] The millimeter-wave signal data obtained after deletion processing is determined as the second dynamic target signal data.

[0099] As an example of this application, the fusion module is used for:

[0100] Acquire second laser signal data, which is a frame of laser signal data adjacent to the first laser signal data;

[0101] The difference between the first laser signal data and the second laser signal data is used as the second dynamic target signal data.

[0102] As an example of this application, the feature dimension space includes angular features and distance features, and the output module is used for:

[0103] Obtain the angle of the target in the first dynamic target signal data, and obtain the distance of the target in the second dynamic target signal data;

[0104] The angle and the distance are determined as the fused data.

[0105] As an example of this application, the output module is also used for:

[0106] Output the distance and angle of the static target in the first laser signal data.

[0107] As an example of this application, the synchronization module is also used for:

[0108] Each of the multiple sensor sampling signals is subjected to regularization processing.

[0109] The sampled signals of each sensor after regularization are synchronized to the feature dimension space.

[0110] As an example of this application, the synchronization module is also used for:

[0111] The multiple detection sensors are synchronized in time and in space. The time synchronization refers to triggering the operation of other detection sensors in the multiple sensors by one of the multiple sensors. The spatial synchronization refers to determining the spatial transformation matrix by multiple detections of the same test target by the multiple detection sensors.

[0112] As an example of this application, the fusion module is used for:

[0113] When the sampling step sizes of the multiple detection sensors are not the same, the sampling step sizes of the multiple signal data are aligned by interpolation.

[0114] The multiple signal data that have undergone alignment processing are then fused.

[0115] As an example of this application, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor;

[0116] The synchronization module is used for:

[0117] The sampling signal of the lidar sensor is converted from the first feature dimension space to the feature dimension space. The first feature dimension space includes the feature dimensions of frame rate feature, scan beam feature, and time feature. The feature dimension space includes the feature dimensions of velocity feature, angle feature, and distance feature.

[0118] The sampling signal of the millimeter-wave radar sensor is converted from the second feature dimension space to the feature dimension space, wherein the second feature dimension space includes chirp features, radiating antenna features, and time features.

[0119] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method described in any of the first aspects above.

[0120] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored on the computer-readable storage medium, and when executed by a processor, the instructions implement the method described in any one of the first aspects above.

[0121] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method described in any of the first aspects above.

[0122] The beneficial effects of the technical solutions provided in this application are:

[0123] The process involves acquiring sampling signals from multiple detection sensors operating at different frequencies within the same detection range at the same time. Each sensor's sampling signal is then synchronized to the same feature dimension space, resulting in multiple signal data sets. This feature dimension space is configured based on the detection dimension of each sensor and the system output requirements. These multiple signal data sets are then fused to obtain a spatial signal description matrix, which describes the signal echo quality of the target. Based on this spatial signal description matrix, fused data is output. This data fusion based on the original sensor sampling signals improves detection robustness and avoids missed detections that might occur with fusion based on point cloud data or target detection results, thus enhancing the accuracy of the final fused data. Attached Figure Description

[0124] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0125] Figure 1 This is a flowchart illustrating a method for multi-sensor signal fusion according to an exemplary embodiment;

[0126] Figure 2 This is a flowchart illustrating a method for multi-sensor signal fusion according to another exemplary embodiment;

[0127] Figure 3 This is a schematic diagram illustrating a sampling signal according to an exemplary embodiment;

[0128] Figure 4 This is a flowchart illustrating a method for multi-sensor signal fusion according to another exemplary embodiment;

[0129] Figure 5 This is a flowchart illustrating a method for multi-sensor signal fusion according to another exemplary embodiment;

[0130] Figure 6 This is a schematic diagram of a point cloud map according to an exemplary embodiment;

[0131] Figure 7 This is a schematic diagram of the structure of a multi-sensor signal fusion device according to an exemplary embodiment;

[0132] Figure 8 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment. Detailed Implementation

[0133] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0134] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " indicates "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, to facilitate a clear description of the technical solutions of this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences.

[0135] Before providing a detailed description of the multi-sensor signal fusion method provided in the embodiments of this application, a brief introduction to the terminology involved in the embodiments of this application will be given first.

[0136] LiDAR sensors: These sensors use laser scanning to reconstruct three-dimensional perception information of the surrounding environment within their detection range. Their working principle involves using Time-of-Flight (TOF) ranging and performing Analog-to-Digital (AD) sampling at the receiving photodiode. After sampling at each angle, the amplitude sequence of the sampled signal is output from the network port to obtain the laser sampling signal. LiDAR sensors can achieve fine three-dimensional perception, reaching an angular resolution of 0.2 degrees. However, when visible noise is present in the surrounding environment, the detection capability of the LiDAR sensor will drastically decrease. For example, in weather conditions such as fog, heavy rain, heavy snow, or sandstorms, the diameter of airborne particles is comparable to the wavelength of the LiDAR sensor, and the quantity of particles is sufficient. This causes the laser light scanned by the LiDAR sensor to attenuate and backscatter when it is incident on these particle aggregates. This results in false detection echoes of airborne particle aggregates and reduces the laser transmission power, thus narrowing the detection range of the LiDAR sensor.

[0137] Millimeter-wave radar sensors sense their surroundings by emitting electromagnetic waves. Their working principle involves using modulation methods, including FMCW (Frequency Modulated Continuous Wave), for spatial measurement. They transmit and receive continuously modulated electromagnetic waves with a base frequency of 24 GHz or 77 GHz from a radio frequency antenna. The echo received by the receiving antenna is sampled by an analog-to-digital converter (AD) and stored in a register. After receiving one frame of signal, it is output from the network interface to obtain the millimeter-wave sampled signal. Millimeter-wave radar sensors are less affected by noise, meaning they have strong anti-interference capabilities. However, their detection accuracy is relatively low, achieving an angular resolution of only 1 degree.

[0138] Time synchronization includes hardware time synchronization and software time synchronization. In one embodiment, hardware time synchronization refers to one sensor sending a synchronization trigger signal to the other at fixed intervals, and the other sensor performing a detection operation upon receiving the synchronization trigger signal. Both sensors output sampling signals simultaneously. Software time synchronization is based on the two detection sensors each having a fixed output frame rate, and aligning and shifting each frame output by both sensors on the same time coordinate axis.

[0139] Spatial synchronization: A landmark (such as a metal marker) is set up where each of the multiple detection sensors is visible in its respective wavelength band. Multiple sensors simultaneously sample the space containing the landmark, and the landmark is then rotated and translated in space to align it. Repeating the sampling and rotation / translation operations yields multiple sets of sampled data. Based on each set of sampled data, the marker's position information is calculated, resulting in multiple sets of position information. Using this multiple set of position information, a system of equations is solved to determine the rotation matrix and translation vector, resulting in a spatial transformation matrix. During spatial synchronization, the determined spatial transformation matrix is ​​used to unify the data from multiple detection sensors into the same coordinate system.

[0140] Next, we will briefly introduce the execution subject involved in the embodiments of this application.

[0141] The multi-sensor signal fusion method provided in this application can be executed by an electronic device. This electronic device can be configured with or connected to multiple detection sensors, which operate in different frequency bands. As an example of this application, the multiple detection sensors include lidar sensors and millimeter-wave radar sensors. Exemplarily, the lidar sensor can be, but is not limited to, any one of 8-line lidar, 16-line lidar, 24-line lidar, 32-line lidar, 64-line lidar, and 128-line lidar. The millimeter-wave radar sensor can be, but is not limited to, any one of 77GHz millimeter-wave radar and 24GHz millimeter-wave radar.

[0142] In practice, lidar sensors and millimeter-wave radar sensors can be installed according to actual needs. For example, they can be fixed to roadside markers (horizontal or vertical poles) to detect targets using lidar and millimeter-wave radar sensors respectively. As an example, targets to be detected can include, but are not limited to, vehicles, pedestrians, non-motorized vehicles, and trees.

[0143] Millimeter-wave radar sensors can employ area-array antennas, covering a forward 180° field of view or less. For lidar sensors, this includes a limited forward field of view and a 360° surround field of view. To ensure field-of-view matching between the millimeter-wave radar and lidar sensors, in one example, a forward-scanning lidar sensor paired with a millimeter-wave radar sensor with an area-array antenna array can complete the forward field of view scanning and imaging. For a 360° surround field of view lidar sensor, 3-4 millimeter-wave radar sensors with area-array antenna arrays are used as needed, with partial overlap in the fields of view of multiple millimeter-wave radar sensors. Alternatively, a 360° surround field of view lidar sensor can be paired with a millimeter-wave radar sensor that also has a 360° surround field of view.

[0144] In some embodiments, the electronic device may include, but is not limited to, wearable devices, terminal devices, in-vehicle systems, camera devices, and roadside base stations. For example, wearable devices may include, but are not limited to, smartwatches, smart bracelets, and smart earmuffs. Additionally, terminal devices may include, but are not limited to, mobile phones, tablets, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs).

[0145] Roadside base stations are crucial infrastructure for intelligent transportation vehicle-road cooperation, serving as service stations integrating sensing, computing, and communication capabilities. In one embodiment, a roadside base station can also be referred to as a smart base station or a roadside fusion sensing system.

[0146] After introducing the terms and implementing entities involved in the embodiments of this application, the method for multi-sensor signal fusion provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0147] Please refer to Figure 1 , Figure 1This is a flowchart illustrating a method for multi-sensor signal fusion according to an exemplary embodiment. As an example and not a limitation, this method can be applied to the aforementioned electronic device. The method may include the following steps:

[0148] Step 101: Obtain the sampling signal of each of the multiple detection sensors targeting the same detection range at the same time, resulting in multiple sensor sampling signals. The multiple detection sensors operate at different frequency bands.

[0149] In one embodiment, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor. In another embodiment, the plurality of detection sensors include a millimeter-wave radar sensor and a visible light detection sensor. In yet another embodiment, the plurality of detection sensors include a millimeter-wave radar sensor and an infrared detection sensor.

[0150] As an example of this application, the electronic device performs target detection using multiple configured detection sensors. During the detection process, the electronic device samples the echo from each of the multiple detection sensors to obtain multiple sensor sampling signals.

[0151] As an example of this application, when detecting a target using multiple detection sensors, the multiple detection sensors can also be synchronized in time and space.

[0152] Time synchronization refers to the process where one of the multiple sensors triggers the operation of the other sensors.

[0153] As an example of this application, taking multiple detection sensors, including a lidar sensor and a millimeter-wave radar sensor, as an example, the specific implementation of time synchronization can include: during the rotation and scanning process of the lidar sensor, whenever the lidar sensor's motor passes its own zero point, it sends a synchronization trigger signal to the millimeter-wave radar sensor. Upon receiving the synchronization trigger signal, the millimeter-wave radar sensor begins to transmit continuous electromagnetic waves. Simultaneously, the receiving antenna of the millimeter-wave radar sensor begins to receive the echo of the electromagnetic waves. After the transmission ends, the reception of the echo also ends, and the lidar sensor has also rotated and scanned the same field of view. In this way, the sampling signals of the two sensors are synchronized in time.

[0154] It should be noted that the above time synchronization is illustrated using the example of a lidar sensor triggering a millimeter-wave radar sensor. In another embodiment, time synchronization can also be achieved by the millimeter-wave radar sensor triggering the lidar sensor. For example, the millimeter-wave radar sensor sends a synchronization trigger signal to the lidar sensor each time it transmits continuous electromagnetic waves. After receiving the synchronization trigger signal, the lidar sensor begins to perform the detection operation.

[0155] Spatial synchronization refers to determining a spatial transformation matrix through multiple detections of the same test target by multiple sensors. For example, after acquiring temporally synchronized laser and millimeter-wave sampling signals, spatial synchronization processing can be performed on the laser and millimeter-wave sampling signals based on the spatial transformation matrix. Subsequent operations are then performed based on the spatially synchronized laser and millimeter-wave sampling signals.

[0156] In practice, because the frequency bands and reflectivities of the various detection sensors are different, the signal amplitudes of the samples from multiple sensors may vary significantly. Therefore, regularization processing can be applied to the sensor sampling signals of each detection sensor to transform the signal amplitudes of each sensor's sampling signals into a uniform amplitude range.

[0157] In one embodiment, the sampling signals of each sensor can be regularized using the following formula (4):

[0158] , (4)

[0159] in, This represents the amplitude of the signal after regularization. This represents the original signal amplitude. This represents the minimum signal amplitude in a frame of probe data. This represents the maximum signal amplitude in a frame of probe data.

[0160] Subsequently, the electronic device performs subsequent steps based on time synchronization, spatial synchronization, and regularized sensor sampling signals.

[0161] Step 102: Synchronize each sensor sampling signal from the multiple sensor sampling signals to the same feature dimension space to obtain multiple signal data. The feature dimension space is set according to the detection dimension of each detection sensor and the system output requirements.

[0162] The detection dimension of a detection sensor is determined by its own characteristics. For example, if the detection sensor is a lidar sensor, then the detection dimension of the lidar sensor (here referred to as the first feature dimension space) includes the following feature dimensions: frame quantity feature, scan beam feature, time feature, and amplitude feature. Among them, the frame quantity feature means that the laser sampling signal corresponds to multiple data frames; the scan beam feature means that the laser sampling signal is obtained by scanning with a scan beam; the time feature means that the laser sampling signal corresponds to a sampling time; and the amplitude feature means that the laser sampling signal has a certain amplitude.

[0163] For example, if the detection sensor is a millimeter-wave radar sensor, then the detection dimension of the millimeter-wave radar sensor (here referred to as the second feature dimension space) includes the following feature dimensions: chirp feature, radiating antenna feature, time feature, and amplitude feature. Here, chirp feature refers to the characteristic that the instantaneous frequency of the pulse changes over time; radiating antenna feature refers to the fact that the millimeter-wave sampling signal is obtained by emitting electromagnetic waves through a radiating antenna; time feature refers to the fact that the millimeter-wave sampling signal corresponds to a sampling time; and amplitude feature refers to the fact that the millimeter-wave sampling signal has a certain amplitude.

[0164] The system output requirements can be set according to actual needs.

[0165] As an example of this application, if the system output requirements include speed, angle, and distance, then the feature dimension space includes speed features, angle features, and distance features. In this case, the electronic device converts the sampling signal from the lidar sensor from the first feature dimension space to this feature dimension space, and converts the sampling signal from the millimeter-wave radar sensor from the second feature dimension space to this feature dimension space.

[0166] Step 103: Perform fusion processing on multiple signal data to obtain a spatial signal description matrix, which can describe the signal echo quality of the target.

[0167] In one embodiment, the electronic device may use weighted, cross-correlation, or binary hypothesis testing methods to fuse multiple signal data to obtain a spatial signal description matrix.

[0168] In one embodiment, considering the problem of discrete digital sampling, the spatial sampling densities of multiple detection sensors may be different. In this case, step 103 specifically includes: when the sampling step sizes of multiple detection sensors are not the same, aligning the sampling step sizes of multiple signal data using interpolation to align the feature spatial sampling step sizes of the multiple detection sensors. Then, fusing the aligned signal data to obtain a spatial signal description matrix.

[0169] Step 104: Output fused data based on the spatial signal description matrix.

[0170] As an example of this application, peak extraction is performed on the fused signal in the spatial signal description matrix, and the fused data is output based on the peak extraction result. For example, the peak extraction result is output as the fused data.

[0171] In this embodiment, sampling signals from multiple detection sensors targeting the same detection range are acquired simultaneously, resulting in multiple sensor sampling signals. These sensors operate at different frequency bands. Each sensor sampling signal is synchronized to the same feature dimension space, yielding multiple signal data. The feature dimension space is set according to the detection dimension of each sensor and the system output requirements. The multiple signal data are then fused to obtain a spatial signal description matrix, which describes the signal echo quality of the target. Based on the spatial signal description matrix, fused data is output. Thus, data fusion based on the original sensor sampling signals improves detection robustness while avoiding missed detections caused by fusion based on point cloud data or target detection results, thereby improving the accuracy of the final generated fused data.

[0172] The following explanation uses multiple detection sensors, including lidar and millimeter-wave radar sensors, as examples. Please refer to... Figure 2 , Figure 2 This is a schematic flowchart illustrating a method for multi-sensor signal fusion according to another exemplary embodiment, which can be applied to the aforementioned electronic device. As an example and not a limitation, the method may include the following implementation steps:

[0173] Step 201: Acquire the sampling signals of the lidar sensor and the millimeter-wave radar sensor at the same time and for the same detection range to obtain the laser sampling signal and the millimeter-wave sampling signal.

[0174] Its implementation can be found in the above. Figure 1 Step 101 in the illustrated embodiment.

[0175] Step 202: Synchronize the laser sampling signal and the millimeter-wave sampling signal to the same feature dimension space respectively to obtain the first laser signal data and the millimeter-wave signal data.

[0176] In one example, the sampling signal from a lidar sensor is transformed from a first feature dimension space to a second feature dimension space, where the first feature dimension space includes frame rate features, scan beam features, and time features. The second feature dimension space includes velocity features, angle features, and distance features. Similarly, the sampling signal from a millimeter-wave radar sensor is transformed from a second feature dimension space to a third feature dimension space, where the second feature dimension space includes chirp features, radiating antenna features, and time features.

[0177] Step 203: Based on each signal data in the multiple signal data, determine at least one evaluation parameter that can evaluate the signal echo quality of the target, and obtain multiple sets of evaluation parameters. Each set of evaluation parameters includes at least one evaluation parameter.

[0178] In this embodiment, the multiple signal data include first laser signal data and millimeter wave signal data.

[0179] As an example of this application, the specific implementation of step 203 may include: determining the millimeter-wave peak position and millimeter-wave echo energy corresponding to the target based on millimeter-wave signal data, and obtaining a set of evaluation parameters. Based on the first laser signal data, determining the echo leading edge position and laser peak saturation corresponding to the target, and obtaining another set of evaluation parameters. The echo leading edge position refers to the position detected by the laser radar sensor when the target is first detected, and the laser peak saturation is used to indicate the confidence level that the target is a real target.

[0180] The position of the millimeter-wave peak is the location where the target has the highest probability of appearing; in other words, the target is most likely to appear at the millimeter-wave peak position. Mathematically speaking, the millimeter-wave peak position is an unbiased estimate of the target's location.

[0181] As an example of this application, a first angular distance map can be determined based on millimeter-wave signal data, and then the position of the millimeter-wave peak can be determined based on the first angular distance map. In one example, the first angular distance map is a spectrum of the millimeter-wave sampled signal. Exemplarily, the millimeter-wave sampled signal can be processed by 3D-FFT (Fast Fourier Transform) to obtain the first angular distance map. The horizontal axis of the first angular distance map is angle, and the vertical axis is distance, corresponding to the spatial sampled signal data in polar coordinates.

[0182] It should be noted that the above description only uses 3D-FFT processing of the millimeter-wave sampled signal to obtain the first angular distance map as an example. In another embodiment, wavelet transform processing can also be performed on the millimeter-wave sampled signal to determine the first angular distance map. In yet another embodiment, subspace transform processing can also be performed on the millimeter-wave sampled signal to determine the first angular distance map.

[0183] As an example of this application, based on the first angle-distance map, the millimeter-wave peak of the target is found using 2D-CFAR (Two Dimensional-Constant False-Alarm Rate) to determine the angle and distance corresponding to the millimeter-wave peak, and then the position of the millimeter-wave peak is determined based on this angle and distance. For example, the position of the millimeter-wave peak can be determined based on this angle and distance using the following formula (5):

[0184] , (5)

[0185] in, The x-coordinate representing the position of the millimeter wave peak. The vertical coordinate representing the position of the millimeter wave peak. Indicates distance, Indicates angle.

[0186] When determining the millimeter-wave echo energy, the electronic device integrates the echo value of the target based on the first angular distance map to determine the millimeter-wave echo energy of the target.

[0187] Electronic devices use millimeter-wave peak position and millimeter-wave echo energy as a set of evaluation parameters.

[0188] As an example of this application, the electronic device determines a second angular distance map based on the first laser signal data, and then determines the echo front position and laser peak saturation corresponding to the target based on the second angular distance map.

[0189] As an example of this application, the electronic device can directly construct a second angular distance map based on the first laser signal data. Similarly, the horizontal axis of the second angular distance map is the angle, and the vertical axis is the distance.

[0190] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of an angular distance map according to an exemplary embodiment, wherein the white highlighted area represents the detection data of the lidar sensor. At the position of the value 20 on the vertical axis, a straight line from left to right represents the detection data of the millimeter-wave radar sensor.

[0191] During laser scanning, when a target is initially detected, the electronic equipment determines its position based on the target's echo; this position is referred to as the echo leading edge position. For example, please refer to... Figure 3 Assuming the goal is Figure 3 In the image 31, the position of the echo leading edge of target 31 is as follows: Figure 3As shown in Figure 32. Since the lidar sensor uses the TOF method for ranging, this embodiment extracts the echo leading edge position and uses it as an unbiased estimate of the target's position.

[0192] As an example of this application, based on the second angle distance map, the waveform of each angle of the lidar sensor is extracted using 1D-CFAR (One Dimensional-Constant False-Alarm Rate) algorithm, and then the leading edge position of the peak is determined to obtain the echo leading edge position.

[0193] Among them, laser peak saturation is used to indicate the characteristics of laser echo and can be used as an evaluation coefficient for laser echo quality.

[0194] As an example of this application, the specific implementation of determining the laser peak saturation of a target detected by a lidar sensor based on a second angular distance map may include: obtaining the laser echo value and laser echo pulse width corresponding to the target based on the second angular distance map; integrating the laser echo value to obtain the laser echo energy; and dividing the laser echo energy by the laser echo pulse width as the laser peak saturation corresponding to the target.

[0195] In one embodiment, the laser echo pulse width corresponding to the target is determined by a dynamic thresholding method based on a second angular distance map.

[0196] It should be noted that the above method for determining the laser peak saturation of a target detected by a lidar sensor based on the second angular distance map is merely exemplary. In another embodiment, its specific implementation may further include: obtaining the laser echo value and laser echo pulse width corresponding to the target based on the second angular distance map; integrating the laser echo value to obtain the laser echo energy; determining the theoretical laser echo energy under the laser echo pulse width; and using the value obtained by dividing the laser echo energy by the theoretical laser echo energy as the laser peak saturation corresponding to the target.

[0197] Step 204: Perform signal fusion processing based on multiple sets of evaluation parameters to obtain the spatial signal description matrix.

[0198] As an example of this application, the specific implementation of step 204 may include: determining the variance of the millimeter-wave peak position based on the millimeter-wave peak position and the millimeter-wave echo energy, obtaining a first variance, which is used to indicate the probability distribution of the target at the millimeter-wave peak position; determining the variance of the echo leading edge position based on the echo leading edge position and the laser peak saturation, obtaining a second variance, which is used to indicate the probability distribution of the target at the echo leading edge position; and determining the spatial signal description matrix based on the millimeter-wave peak position, the echo leading edge position, the first variance, and the second variance.

[0199] As an example rather than a limitation, the variance of the millimeter-wave peak position is determined by the following formula (6) based on the millimeter-wave peak position and the millimeter-wave echo energy:

[0200] (6)

[0201] in, The first variance, For millimeter wave peak locations, including ( ), This represents the millimeter-wave echo energy. A and B are adjustable parameters that can be set according to actual needs.

[0202] As an example rather than a limitation, the variance of the echo leading edge position is determined by the following formula (7) based on the echo leading edge position and the laser peak saturation:

[0203] (7)

[0204] in, The second variance, The position of the echo leading edge includes ( ), This indicates the saturation of the laser peak.

[0205] As an example rather than a limitation, the spatial signal description matrix is ​​determined based on the millimeter wave peak position, echo leading edge position, first variance, and second variance using the following formulas (1) to (3):

[0206] (1)

[0207] (2)

[0208] (3)

[0209] in, For joint positions, including ( ), For millimeter wave peak locations, including ( ), The position of the echo leading edge includes ( ), The first variance, For the second variance, ( ) is a variable. Let V be the joint confidence level, and let V be the joint confidence level distribution.

[0210] Electronic devices can determine the joint confidence distribution map based on V. Then, the joint confidence distribution map is transformed to obtain the spatial signal description matrix, that is, the joint confidence distribution map is represented by the spatial signal description matrix.

[0211] It should be noted that steps 203 to 204 above are an exemplary implementation of fusing multiple signal data.

[0212] Step 205: Output fused data based on the spatial signal description matrix.

[0213] As an example of this application, step 205 may specifically include: determining the joint positions where the joint confidence level is greater than a confidence threshold from the spatial signal description matrix; and outputting the joint positions where the joint confidence level is greater than the confidence threshold and the corresponding joint confidence levels as fused data.

[0214] The confidence threshold can be set by the user according to actual needs, or it can be set by default by the electronic device. This application embodiment does not limit this.

[0215] Since the joint location is the location where the target has the highest probability of being detected by both the millimeter-wave radar sensor and the lidar sensor, when the joint confidence of the joint location is greater than the confidence threshold, it means that the joint location has a high confidence level as the location where the target has the highest probability of being detected. Therefore, the electronic device outputs the joint location and its joint confidence as fused data.

[0216] It should be noted that the specific implementation of outputting fused data based on the spatial signal description matrix is ​​merely exemplary. In another embodiment, the output fused data may also include other detection data of the target. In one embodiment, step 205 may further include determining joint locations from the spatial signal description matrix where the joint confidence score is greater than a confidence threshold. The joint locations where the joint confidence score is greater than the confidence threshold, the corresponding joint confidence score, and the velocity are output as fused data. This velocity refers to the velocity detected by the millimeter-wave radar sensor. That is, the output point cloud data may also include the velocity detected by the millimeter-wave radar sensor. In another embodiment, each joint location and the joint confidence score of each joint location in the spatial signal description matrix may also be output as point cloud data. When using the point cloud data subsequently, filtering can be performed based on the joint confidence score of each joint location.

[0217] In one embodiment, to facilitate users in viewing the location distribution of targets detected by the millimeter-wave radar sensor and the lidar sensor respectively, a confidence map of the millimeter-wave radar sensor signal and a confidence map of the lidar sensor signal can also be plotted.

[0218] For example, the confidence distribution of the signal from a millimeter-wave radar sensor can be determined using the following formulas (8) and (9):

[0219] (8)

[0220] (9)

[0221] in, This represents the confidence distribution of signals from a millimeter-wave radar sensor.

[0222] Thus, based on the above... This allows for the creation of a confidence map of the millimeter-wave radar sensor signal.

[0223] For example, the confidence distribution of the lidar sensor signal can be determined using the following formulas (10) and (11):

[0224] (10)

[0225] (11)

[0226] in, This represents the confidence distribution of the signal from the lidar sensor. It is the ranging distance of the lidar sensor.

[0227] Thus, based on the above... This allows you to plot the confidence level of the LiDAR sensor signal.

[0228] In this embodiment, millimeter-wave sampling signals and laser sampling signals are acquired simultaneously within the same detection range. These signals are synchronized to the same feature dimension space. A first angular distance map is obtained based on the synchronized millimeter-wave sampling signals, and a second angular distance map is obtained based on the synchronized laser sampling signals. The first and second angular distance maps are fused to obtain a spatial signal description matrix. This matrix describes the joint position and joint confidence of the target, where the joint position represents the location with the highest probability of target occurrence. Thus, fusion processing based on the original sampling signals, compared to fusion based on point cloud data or detection results, enhances the sampling signals of ambiguous but real targets, avoiding the false rejection of such targets. Furthermore, fused data is output based on the fused spatial signal description matrix. Since the spatial signal description matrix describes the joint position and joint confidence of the target, the accuracy of the final generated fused data is improved.

[0229] The following explanation uses multiple detection sensors, including lidar and millimeter-wave radar sensors, as examples. Please refer to... Figure 4 , Figure 4 This is a schematic flowchart illustrating a method for multi-sensor signal fusion according to another exemplary embodiment, which can be applied to the aforementioned electronic device. As an example and not a limitation, the method may include the following implementation steps:

[0230] Steps 404 to 402 can be found in steps 201 to 202 above.

[0231] Step 403: Perform location correlation based on multiple signal data.

[0232] In this embodiment, the multiple signal data include first laser signal data and millimeter wave signal data.

[0233] In order to determine which signals in the first laser signal data and the millimeter-wave signal data belong to the same target, the electronic device performs position association processing based on the first laser signal data and the millimeter-wave signal data.

[0234] In one example of this application, a bipartite graph matching method can be used for location association. Exemplarily, the bipartite graph matching method can be the Hungarian algorithm, etc., and this application does not limit it to this embodiment.

[0235] In one embodiment, in addition to dynamic targets, static targets may also exist within the detection range. However, since the echoes of dynamic targets detected by millimeter-wave radar sensors are easily overwhelmed by the echoes of static targets, as an example of this application, the electronic device can filter out static targets from the sensor signal data to retain the signal data of dynamic targets for subsequent fusion processing.

[0236] In one example, the electronic device filters out the signal data of static targets detected by the lidar sensor from the first laser signal data to obtain the first dynamic target signal data, and filters out the signal data of static targets detected by the millimeter-wave radar sensor from the millimeter-wave signal data to obtain the second dynamic target signal data.

[0237] In other words, in order to fuse the signal data of dynamic targets, the electronic equipment filters out the signal data of static targets in the millimeter-wave signal data of the millimeter-wave radar sensor, and filters out the signal data of static targets in the first laser signal data of the lidar sensor.

[0238] In one embodiment, the millimeter-wave signal data includes velocity. Accordingly, a specific implementation of filtering out the signal data of static targets detected by the millimeter-wave radar sensor from the millimeter-wave signal data to obtain the second dynamic target signal data may include: determining signal data with velocities less than a velocity threshold from the millimeter-wave signal data; deleting signal data with velocities less than the velocity threshold from the millimeter-wave signal data; and determining the millimeter-wave signal data obtained after deletion as the second dynamic target signal data.

[0239] The speed threshold can be set by the user according to actual needs, or it can be set by default by the electronic device. This application embodiment does not limit this.

[0240] It's easy to understand that if a target's speed is less than a speed threshold, it indicates that the target may be stationary or only slightly moving. In this case, the target can be identified as a static target. The electronic device removes the target's signal data from the millimeter-wave signal data. In this way, the signal data of static targets can be filtered out from the millimeter-wave signal data, leaving the signal data of dynamic targets detected by the millimeter-wave radar sensor, which is the second dynamic target signal data.

[0241] It should be noted that this description takes the determination of a target with a speed less than a speed threshold as a static target as an example. In another embodiment, a target with a speed less than or equal to a speed threshold can also be determined as a static target. This application does not limit this aspect.

[0242] In one embodiment, filtering out the signal data of static targets detected by the lidar sensor from the first laser signal data to obtain the first dynamic target signal data may include: acquiring second laser signal data, where the second laser signal data is a frame of laser signal data adjacent to the first laser signal data; and using the signal data difference obtained by subtracting the first laser signal data from the second laser signal data as the first dynamic target signal data.

[0243] As one example, the second laser signal data can be the laser signal data of the previous frame of the first laser signal data. As another example, the second laser signal data can also be the laser signal data of the next frame of the first laser signal data.

[0244] It's easy to understand that if a target is stationary, subtracting the signal data from two adjacent frames of the lidar sensor will filter out the static target's signal data. Therefore, by subtracting the echoes from adjacent frames of the first lidar signal data, the signal data of the dynamic target detected by the lidar sensor can be determined, thus obtaining the first dynamic target signal data.

[0245] It should be noted that this explanation only uses the example of subtracting the second laser signal data from the adjacent frame of laser signal data to determine the first dynamic target signal data. In another embodiment, laser signal data separated from the first laser signal data by a preset number of frames can also be obtained to obtain the third laser signal data. Then, the third laser signal data is subtracted from the first laser signal data to determine the first dynamic target signal data. The preset number can be set according to actual needs. For example, the laser signal data of the frame preceding or following the second laser signal data can also be obtained to obtain the third laser signal data. For instance, if the second laser signal data is the previous frame of the first laser signal data, the previous frame of the second laser signal data can be used as the third laser signal data; similarly, if the second laser signal data is the next frame of the first laser signal data, the next frame of the second laser signal data can be used as the third laser signal data. This application does not limit this aspect.

[0246] As an example of this application, a specific implementation of location association based on multiple signal data includes: performing location association based on first dynamic target signal data and second dynamic target signal data. The association method can employ bipartite graph matching.

[0247] Step 404: Based on the location association results, determine the target's signal data from each of the multiple signal data to obtain multiple target signal data of the target.

[0248] By establishing location correlation, it's possible to identify targets detected simultaneously by both lidar and millimeter-wave radar sensors, or in other words, to determine if they are detecting the same target. Then, the associated target signal data is extracted from the signal data of each sensor, resulting in multiple target signal data sets.

[0249] The number of associated targets may be one or more. When there are multiple associated targets, the signal data of each associated target can be fused according to the following method.

[0250] It should be noted that if multiple detection sensors include millimeter-wave radar sensors and visible light sensors, the same target detected by both can be determined through sampling position matching. Then, the signal data of the same target is determined from the millimeter-wave signal data of the millimeter-wave radar sensor, and the signal data of the same target is determined from the visible light image of the visible light sensor, thus obtaining multiple signal data of the target. Similarly, if multiple detection sensors include millimeter-wave radar sensors and infrared sensors, the same procedure applies, which will not be repeated here.

[0251] Step 405: Perform fusion processing on multiple target signal data to obtain a spatial signal description matrix.

[0252] In one embodiment, multiple target signal data are arranged in a certain manner to obtain a spatial signal description matrix. For example, target signal data belonging to the first dynamic target signal data are arranged in the first column, and target signal data belonging to the second dynamic target signal data are arranged in the second column to obtain the spatial signal description matrix.

[0253] Step 406: Output fused data based on the spatial signal description matrix.

[0254] As an example of this application, step 406 may specifically include: acquiring the angle of the target in the first dynamic target signal data and acquiring the distance of the target in the second dynamic target signal data. The angle and distance are then used to determine the fused data.

[0255] Because millimeter-wave radar sensors have low angular resolution, the target angle of arrival measurement can be provided by lidar sensors. Therefore, please refer to... Figure 5During data fusion, the angle from the first dynamic target signal data detected by the lidar sensor is used as the output. Since the ranging information from the millimeter-wave radar sensor is more accurate, and the ranging of the lidar sensor is affected by noise, the target ranging information is provided by the millimeter-wave radar sensor. Therefore, the distance from the second dynamic target signal data from the millimeter-wave radar sensor is used as the output, thus obtaining the fused data.

[0256] As an example of this application, the distance and angle of the static target in the first laser signal data are output. That is, for static targets, the detection results of the lidar sensor can be directly used as the output.

[0257] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram of signal data according to an exemplary embodiment, wherein the horizontal axis represents angle, from -10 degrees to 10 degrees, and the vertical axis represents distance, from 0 meters to 80 meters. Figure 3 The white area (using white as an example, but other colors like yellow) represents the signal data from the lidar sensor, while the line from left to right at a distance of approximately 20 meters represents the signal data from the millimeter-wave radar sensor. This means the millimeter-wave radar sensor detected a target at 20 meters but doesn't know its exact angle, while the lidar sensor can clearly detect the target's outline originating from approximately 5 degrees. Therefore, the distance to targets at approximately 5 degrees and 20 meters can be determined by the millimeter-wave radar's ranging results, while the lidar sensor provides the specific angle from which the target originates at approximately 5 degrees. Furthermore, targets not detected by the millimeter-wave radar are static targets that have been filtered out; their distance and angle can be obtained from the lidar sensor's detection results. Thus, we can obtain... Figure 6 The point cloud diagram shown.

[0258] As another example of this application, the specific implementation of step 406 may further include: acquiring the angle of the target in the first dynamic target signal data, and acquiring the distance and velocity of the target in the second dynamic target signal data. The angle, distance, and velocity of the target are then determined as fused data.

[0259] In this embodiment, sampling signals from multiple detection sensors targeting the same detection range are acquired simultaneously, resulting in multiple sensor sampling signals. These sensors operate at different frequency bands. The multiple sensor sampling signals are then synchronized to the same feature dimension space, yielding multiple signal data. Based on these multiple signal data of the target, the data is fused to obtain fused data. This data fusion based on the original sensor sampling signals improves detection robustness while avoiding missed detections caused by fusion based on point cloud data or target detection results, thus improving the accuracy of the final generated fused data.

[0260] It should be noted that when multiple detection sensors, including millimeter-wave radar sensors and visible light sensors, are used to fuse multiple signal data based on the target, the signal data detected by the millimeter-wave radar sensor can be used as an additional channel in the signal data detected by the visible light sensor to obtain fused data. For example, the distance from the millimeter-wave radar sensor's signal data can be used as an additional channel in the visible light image; alternatively, the distance and angle from the millimeter-wave radar sensor's signal data can be used as two additional channels in the visible light image, respectively.

[0261] Similarly, when multiple detection sensors, including millimeter-wave radar sensors and infrared sensors, are used to fuse multiple signal data based on a target, the signal data detected by the millimeter-wave radar sensor can be used as an additional channel in the signal data detected by the infrared sensor to obtain fused data. For example, the distance from the millimeter-wave radar sensor's signal data can be used as an additional channel in the infrared image; or, for instance, the distance and angle from the millimeter-wave radar sensor's signal data can be used as two additional channels in the infrared image, respectively.

[0262] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0263] Based on the multi-sensor signal fusion methods provided in the above embodiments, please refer to... Figure 7 , Figure 7 This is a schematic diagram illustrating the structure of a multi-sensor signal fusion device according to an exemplary embodiment. This device can be implemented as part or all of the aforementioned electronic device by software, hardware, or a combination of both. The device may include:

[0264] The acquisition module 710 is used to acquire the sampling signal of each of the multiple detection sensors targeting the same detection range at the same time, thereby obtaining multiple sensor sampling signals, wherein the multiple detection sensors operate in different frequency bands;

[0265] The synchronization module 720 is used to synchronize each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space to obtain multiple signal data, wherein the feature dimension space is set according to the detection dimension of each detection sensor and the system output requirements;

[0266] The fusion module 730 is used to fuse the multiple signal data to obtain a spatial signal description matrix, which can describe the signal echo quality of the target.

[0267] Output module 740 is used to output fused data based on the spatial signal description matrix.

[0268] As an example of this application, the fusion module 730 is used for:

[0269] Based on each of the plurality of signal data, at least one evaluation parameter capable of evaluating the signal echo quality of the target is determined, resulting in a plurality of evaluation parameter sets, each of the plurality of evaluation parameter sets including the at least one evaluation parameter;

[0270] Signal fusion processing is performed based on the aforementioned set of multiple evaluation parameters.

[0271] As an example of this application, the plurality of signal data includes first laser signal data and millimeter-wave signal data; the fusion module 730 is used for:

[0272] The step involves determining at least one evaluation parameter capable of assessing the signal echo quality of the target based on each of the plurality of signal data, resulting in a set of multiple evaluation parameters, including:

[0273] Based on the millimeter-wave signal data, the position of the millimeter-wave peak and the millimeter-wave echo energy corresponding to the target are determined, and a set of evaluation parameters is obtained.

[0274] Based on the first laser signal data, the echo leading edge position and laser peak saturation corresponding to the target are determined to obtain another set of evaluation parameters. The echo leading edge position refers to the position detected by the lidar sensor when the target is first detected by the lidar sensor, and the laser peak saturation is used to indicate the confidence level that the target is a real target.

[0275] As an example of this application, the fusion module 730 is used for:

[0276] Based on the millimeter wave peak position and the millimeter wave echo energy, the variance of the millimeter wave peak position is determined to obtain a first variance, which is used to indicate the probability distribution of the target at the millimeter wave peak position.

[0277] Based on the echo leading edge position and the laser peak saturation, the variance of the echo leading edge position is determined to obtain a second variance, which is used to indicate the probability distribution of the target at the echo leading edge position.

[0278] The spatial signal description matrix is ​​determined based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance.

[0279] As an example of this application, the fusion module 730 is used for:

[0280] Based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance, the spatial signal description matrix is ​​determined using the following formulas (1) to (3):

[0281] (1)

[0282] (2)

[0283] (3)

[0284] Among them, the include( ), the The millimeter wave peak position includes ( ), the The position of the leading edge of the echo includes ( ), the For the first variance, the For the second variance, the ( ) is a variable, the V represents the joint confidence level, and V is the joint confidence level distribution.

[0285] As an example of this application, the output module 740 is used for:

[0286] From the spatial signal description matrix, determine the joint positions where the joint confidence level is greater than the confidence threshold;

[0287] The joint positions with a joint confidence level greater than a confidence threshold and their corresponding joint confidence levels are output as the fused data.

[0288] As an example of this application, the fusion module 730 is used for:

[0289] Location association is performed based on the multiple signal data;

[0290] Based on the location association results, the signal data of the target is determined from each of the multiple signal data, thereby obtaining multiple target signal data of the target;

[0291] The multiple target signal data are fused together.

[0292] As an example of this application, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor, and the plurality of signal data includes first laser signal data and millimeter-wave signal data;

[0293] The fusion module 730 is used to: filter out the signal data of the static target detected by the lidar sensor from the first laser signal data to obtain the first dynamic target signal data, and filter out the signal data of the static target detected by the millimeter-wave radar sensor from the millimeter-wave signal data to obtain the second dynamic target signal data;

[0294] Accordingly, the specific implementation of the fusion module 730 in performing position association based on the multiple signal data includes: performing position association based on the first dynamic target signal data and the second dynamic target signal data.

[0295] As an example of this application, the millimeter-wave signal data includes velocity, and the fusion module 730 is used for:

[0296] Delete the signal data whose velocity is less than the velocity threshold from the millimeter wave signal data;

[0297] The millimeter-wave signal data obtained after deletion processing is determined as the second dynamic target signal data.

[0298] As an example of this application, the fusion module 730 is used for:

[0299] Acquire second laser signal data, which is a frame of laser signal data adjacent to the first laser signal data;

[0300] The difference between the first laser signal data and the second laser signal data is used as the second dynamic target signal data.

[0301] As an example of this application, the feature dimension space includes angular features and distance features, and the output module 740 is used for:

[0302] Obtain the angle of the target in the first dynamic target signal data, and obtain the distance of the target in the second dynamic target signal data;

[0303] The angle and the distance are determined as the fused data.

[0304] As an example of this application, the output module 740 is also used for:

[0305] Output the distance and angle of the static target in the first laser signal data.

[0306] As an example of this application, the synchronization module 720 is also used for:

[0307] Each of the multiple sensor sampling signals is subjected to regularization processing.

[0308] The sampled signals of each sensor after regularization are synchronized to the feature dimension space.

[0309] As an example of this application, the synchronization module 720 is also used for:

[0310] The multiple detection sensors are synchronized in time and in space. The time synchronization refers to triggering the operation of other detection sensors in the multiple sensors by one of the multiple sensors. The spatial synchronization refers to determining the spatial transformation matrix by multiple detections of the same test target by the multiple detection sensors.

[0311] As an example of this application, the fusion module 730 is used for:

[0312] When the sampling step sizes of the multiple detection sensors are not the same, the sampling step sizes of the multiple signal data are aligned by interpolation.

[0313] The multiple signal data that have undergone alignment processing are then fused.

[0314] As an example of this application, the plurality of detection sensors include a lidar sensor and a millimeter-wave radar sensor;

[0315] The synchronization module 720 is used for:

[0316] The sampling signal of the lidar sensor is converted from the first feature dimension space to the feature dimension space. The first feature dimension space includes the feature dimensions of frame rate feature, scan beam feature, and time feature. The feature dimension space includes the feature dimensions of velocity feature, angle feature, and distance feature.

[0317] The sampling signal of the millimeter-wave radar sensor is converted from the second feature dimension space to the feature dimension space, wherein the second feature dimension space includes chirp features, radiating antenna features, and time features.

[0318] In this embodiment, sampling signals from multiple detection sensors targeting the same detection range are acquired simultaneously, resulting in multiple sensor sampling signals. These sensors operate at different frequency bands. Each sensor sampling signal is synchronized to the same feature dimension space, yielding multiple signal data. The feature dimension space is set according to the detection dimension of each sensor and the system output requirements. The multiple signal data are then fused to obtain a spatial signal description matrix, which describes the signal echo quality of the target. Based on the spatial signal description matrix, fused data is output. Thus, data fusion based on the original sensor sampling signals improves detection robustness while avoiding missed detections caused by fusion based on point cloud data or target detection results, thereby improving the accuracy of the final generated fused data.

[0319] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 (Only one is shown in the diagram), memory 81, and computer program 82 stored in said memory 81 and executable on said at least one processor 80, wherein said processor 80 executes said computer program 82 to implement the steps in any of the above method embodiments.

[0320] The electronic device 8 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This electronic device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0321] The processor 80 may be a CPU (Central Processing Unit), or it may be other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0322] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may be an external storage device of the electronic device 8, such as a plug-in hard disk, SMC (Smart Media Card), SD (Secure Digital) card, flash card, etc., equipped on the electronic device 8. Furthermore, the memory 81 may include both internal and external storage units of the electronic device 8. The memory 81 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0323] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0324] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0325] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for multi-sensor signal fusion, characterized in that, The method includes: The sampling signals of each of the multiple detection sensors targeting the same detection range at the same time are acquired to obtain multiple sensor sampling signals, wherein the multiple detection sensors operate in different frequency bands; Each sensor sampling signal from the plurality of sensor sampling signals is synchronized to the same feature dimension space to obtain multiple signal data, wherein the feature dimension space is set according to the detection dimension of each detection sensor and the system output requirements, and the multiple signal data includes first laser signal data and millimeter wave signal data; Based on each of the plurality of signal data, at least one evaluation parameter capable of evaluating the signal echo quality of the target is determined, resulting in a plurality of evaluation parameter sets, each of the plurality of evaluation parameter sets including the at least one evaluation parameter; Based on the multiple sets of evaluation parameters, signal fusion processing is performed to obtain a spatial signal description matrix, which can describe the signal echo quality of the target. Based on the spatial signal description matrix, output fused data; Specifically, the step of determining at least one evaluation parameter capable of evaluating the signal echo quality of the target based on each of the plurality of signal data, resulting in a set of multiple evaluation parameters, including: Based on the millimeter-wave signal data, the position of the millimeter-wave peak and the millimeter-wave echo energy corresponding to the target are determined, and a set of evaluation parameters is obtained. Based on the first laser signal data, the echo leading edge position and laser peak saturation corresponding to the target are determined to obtain another set of evaluation parameters. The echo leading edge position refers to the position detected by the lidar sensor when the target is first detected by the lidar sensor, and the laser peak saturation is used to indicate the confidence level that the target is a real target.

2. The method as described in claim 1, characterized in that, The signal fusion processing based on the multiple sets of evaluation parameters includes: Based on the millimeter wave peak position and the millimeter wave echo energy, the variance of the millimeter wave peak position is determined to obtain a first variance, which is used to indicate the probability distribution of the target at the millimeter wave peak position. Based on the echo leading edge position and the laser peak saturation, the variance of the echo leading edge position is determined to obtain a second variance, which is used to indicate the probability distribution of the target at the echo leading edge position. The spatial signal description matrix is ​​determined based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance.

3. The method as described in claim 2, characterized in that, The determination of the spatial signal description matrix based on the millimeter-wave peak position, the echo leading edge position, the first variance, and the second variance includes: Based on the millimeter wave peak position, the echo leading edge position, the first variance, and the second variance, the joint confidence distribution is determined by the following formulas (1) to (3); A joint confidence distribution map is determined based on the aforementioned joint confidence distribution; The joint confidence distribution map is transformed to obtain the spatial signal description matrix: （1） （2） （3） Among them, the For joint positions, including ( ), the The millimeter wave peak position includes ( ), the The position of the leading edge of the echo includes ( ), the For the first variance, the For the second variance, the ( ) is a variable, the V represents the joint confidence level, and V is the distribution of the joint confidence level.

4. The method according to any one of claims 1-3, characterized in that, The output of fused data based on the spatial signal description matrix includes: From the spatial signal description matrix, determine the joint positions where the joint confidence level is greater than the confidence threshold; The joint positions with a joint confidence level greater than a confidence threshold and their corresponding joint confidence levels are output as the fused data.

5. The method as described in claim 1, characterized in that, Before synchronizing each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space, the method further includes: Each of the multiple sensor sampling signals is subjected to regularization processing. The step of synchronizing each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space includes: The sampled signals of each sensor after regularization are synchronized to the feature dimension space.

6. The method as described in claim 1, characterized in that, The method further includes: The multiple detection sensors are synchronized in time and in space. The time synchronization refers to triggering the operation of other detection sensors in the multiple sensors by one of the multiple sensors. The spatial synchronization refers to determining the spatial transformation matrix by multiple detections of the same test target by the multiple detection sensors.

7. The method as described in claim 1, characterized in that, The step of determining at least one evaluation parameter capable of assessing the signal echo quality of a target based on each of the plurality of signal data includes: When the sampling step sizes of the multiple detection sensors are not the same, the sampling step sizes of the multiple signal data are aligned by interpolation. Based on each of the plurality of signal data after alignment processing, at least one evaluation parameter capable of evaluating the signal echo quality of the target is determined.

8. The method as described in claim 1, characterized in that, The plurality of detection sensors include lidar sensors and millimeter-wave radar sensors; The step of synchronizing each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space includes: The sampling signal of the lidar sensor is converted from the first feature dimension space to the feature dimension space. The first feature dimension space includes the feature dimensions of frame rate feature, scan beam feature, and time feature. The feature dimension space includes the feature dimensions of velocity feature, angle feature, and distance feature. The sampling signal of the millimeter-wave radar sensor is converted from the second feature dimension space to the feature dimension space, wherein the second feature dimension space includes chirp features, radiating antenna features, and time features.

9. A device for multi-sensor signal fusion, characterized in that, The device includes: The acquisition module is used to acquire the sampling signal of each of the multiple detection sensors targeting the same detection range at the same time, thereby obtaining multiple sensor sampling signals, wherein the multiple detection sensors operate in different frequency bands; A synchronization module is used to synchronize each sensor sampling signal from the plurality of sensor sampling signals to the same feature dimension space to obtain multiple signal data. The feature dimension space is set according to the detection dimension of each detection sensor and the system output requirements. The multiple signal data includes first laser signal data and millimeter wave signal data. The fusion module is used to determine at least one evaluation parameter that can evaluate the signal echo quality of the target based on each of the multiple signal data, to obtain multiple sets of evaluation parameters, each of the multiple sets of evaluation parameters including the at least one evaluation parameter; and to perform signal fusion processing based on the multiple sets of evaluation parameters to obtain a spatial signal description matrix, which can describe the signal echo quality of the target. The output module is used to output fused data based on the spatial signal description matrix; The fusion module determines at least one evaluation parameter for assessing the signal echo quality of the target based on each of the multiple signal data sets, resulting in a set of multiple evaluation parameters, including: Based on the millimeter-wave signal data, the position of the millimeter-wave peak and the millimeter-wave echo energy corresponding to the target are determined, and a set of evaluation parameters is obtained. Based on the first laser signal data, the echo leading edge position and laser peak saturation corresponding to the target are determined to obtain another set of evaluation parameters. The echo leading edge position refers to the position detected by the lidar sensor when the target is first detected by the lidar sensor, and the laser peak saturation is used to indicate the confidence level that the target is a real target.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images