Acoustic-optical radar data fusion method, device, radar and computer storage medium

Abstract

This invention relates to an acoustic-optical radar data fusion method, apparatus, radar, and computer storage medium, belonging to the field of radar technology. The acoustic-optical radar data fusion method includes: acquiring acoustic reflection signal data and optical reflection signal data collected by a sonar system and a lidar system targeting a target obstacle; determining distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data; determining the fault status of the sonar system and the lidar system based on the distance fluctuation information; determining a reference value and a correction value for acoustic-optical radar data fusion based on the fault status of the sonar system and the lidar system; and fusing the reference value and the correction value to obtain fused acoustic-optical radar data. This invention can ensure the accuracy of environmental perception and adapt to complex environmental requirements.

Description

Technical Field

[0001] This invention relates to the field of radar technology, and in particular to an acoustic-optical radar data fusion method, apparatus, radar, and computer storage medium. Background Technology

[0002] With the development of technologies such as intelligent sensing and autonomous driving, the requirements for environmental sensing technology are becoming increasingly stringent.

[0003] In the field of modern intelligent sensing and navigation technology, the development of radar technology has greatly promoted the progress of robotics and autonomous driving technology. Traditional radar includes lidar and sonar systems. Lidar systems have good single-precision in long-distance environmental perception, but in certain environments, they are affected by many factors, resulting in a significant decrease in detection accuracy or even complete failure, such as in complex environments with dense smoke, underwater, or highly reflective surfaces. Although sonar systems are not as accurate and have lower resolution as lidar, they exhibit stronger detection capabilities at close range and in complex environments, such as in dense smoke, darkness, or underwater scenarios.

[0004] This shows that a single radar detection system cannot meet the complex needs of modern times and cannot achieve accurate perception of the entire environment. Summary of the Invention

[0005] In view of this, it is necessary to provide an acoustic-optical radar data fusion method, device, radar, and computer storage medium to solve the problem that a single radar detection system cannot meet the complex modern needs and cannot achieve accurate perception of the entire environment.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides an acoustic-optical radar data fusion method, comprising: Acquire acoustic and optical reflection signal data collected by sonar and lidar systems against target obstacles; Based on acoustic reflection signal data and optical reflection signal data, determine the distance fluctuation information of the target obstacle collected by the sonar system and lidar system, and determine the fault status of the sonar system and lidar system based on the distance fluctuation information. The baseline and correction values ​​for acoustic-optical radar data fusion are determined based on the fault conditions of the sonar and lidar systems, and the baseline and correction values ​​are fused to obtain the acoustic-optical radar fused data.

[0007] In one possible implementation, acquiring acoustic reflection signal data collected by the sonar system against a target obstacle includes: Acquire multi-angle acoustic reflection signal data collected by the sonar system against the target obstacle, and determine the time delay of the acoustic reflection signal data at each angle; A time-delay summation beamforming algorithm is used to fuse acoustic reflection signal data from each angle based on the time delay of the acoustic reflection signal from each angle, so as to obtain acoustic reflection signal data collected by the sonar system for the target obstacle.

[0008] In one possible implementation, distance fluctuation information of the target obstacle collected by the sonar system and the lidar system is determined based on acoustic reflection signal data and optical reflection signal data, and the fault status of the sonar system and the lidar system is determined based on the distance fluctuation information, including: A preset depth belief network is used to extract the first distance fluctuation characteristics of the target obstacle from the acoustic reflection signal data and the second distance fluctuation characteristics of the target obstacle from the optical reflection signal data; A pre-defined single-class support vector machine is used to classify the first and second range fluctuation features of the target obstacle to determine the fault status of the sonar system and lidar system.

[0009] In one possible implementation, determining the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar and lidar systems includes: When the sonar system malfunctions but the lidar system does not, the light reflection signal data is used as the reference value and the acoustic reflection signal data is used as the correction value. When the lidar system malfunctions but the sonar system does not, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value.

[0010] In one possible implementation, determining the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar and lidar systems includes: When neither the sonar system nor the lidar system malfunctions, calculate the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data. When the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value. When the first variance is greater than the second variance, the light reflection signal data is used as the reference value, and the sound reflection signal data is used as the correction value.

[0011] In one possible implementation, the reference value and the correction value are fused to obtain acoustic-optical radar fused data, including: Calculate the absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data; The fusion parameters are adjusted in real time based on the relationship between the absolute value of the difference and the preset difference threshold. An adaptive Kalman filter algorithm is used to fuse the baseline and correction values ​​based on real-time adjusted fusion parameters to obtain fused acoustic-optical radar data.

[0012] In one possible implementation, the formula for adjusting the fusion parameters is:

[0013] in, and For a pre-set threshold, and , and These are the pre-set basic fusion parameters, and ; The formula for combining the baseline value and the correction value is:

[0014] in, F For audio-visual fusion data, As the baseline value, This is a correction value.

[0015] Secondly, the present invention also provides an acoustic-optical radar data fusion device, characterized in that it comprises: The data acquisition module is used to acquire acoustic reflection signal data and optical reflection signal data collected by the sonar system and lidar system against the target obstacle; The fault diagnosis module is used to determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data, and to determine the fault status of the sonar system and the lidar system based on the distance fluctuation information. The data fusion module is used to determine the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar system and lidar system, and to fuse the baseline and correction values ​​to obtain acoustic-optical radar fused data.

[0016] Thirdly, the present invention also provides an acoustic-optical fusion radar, characterized in that it includes a memory and a processor, wherein... The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the acoustic-optical radar data fusion method described in any of the above embodiments.

[0017] Fourthly, the present invention also provides a computer-readable storage medium, characterized in that it is used to store a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the acoustic-optical radar data fusion method described in any of the above embodiments.

[0018] The beneficial effects of this invention are as follows: The acoustic-optical radar data fusion method provided by this invention collects acoustic reflection signal data and optical reflection signal data of target obstacles through a sonar system and a lidar system. Based on the collected acoustic and optical reflection signal data, it determines the distance fluctuation information of the target obstacles collected by the sonar and lidar systems. Based on this distance fluctuation information, it determines whether the sonar and lidar systems are faulty. Based on the fault status of the sonar and lidar systems, it determines the reference value and correction value for acoustic-optical radar data fusion, and fuses the reference value and the correction value to obtain fused acoustic-optical radar data. Simultaneously using a lidar system and a sonar system for environmental perception effectively avoids the limitations of a single radar system. Furthermore, by analyzing the distance fluctuation information of the target obstacles collected by the sonar and lidar systems to determine whether the sonar and lidar systems are faulty, when one radar system fails, the other radar system can be used to correct the detected target obstacle information, ensuring the accuracy of environmental perception and adapting to complex environmental requirements. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an acoustic-optical radar data fusion method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a method for acquiring acoustic reflection signal data according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a fault diagnosis method provided in an embodiment of the present invention. Figure 4 This invention provides a structural diagram of a depth belief network using stacked restricted Boltzmann machines, as shown in an embodiment of the invention. Figure 5 This is a structural diagram of a single-class support vector machine provided in an embodiment of the present invention; Figure 6 A flowchart illustrating a data fusion method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an acoustic-optical radar data fusion device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an acoustic-optical fusion radar provided in an embodiment of the present invention. Detailed Implementation

[0021] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0022] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] A specific embodiment of the present invention, such as Figure 1 As shown, an acoustic-optical radar data fusion method is disclosed, including: S101, acquire acoustic reflection signal data and optical reflection signal data collected by the sonar system and lidar system against the target obstacle.

[0025] In this embodiment of the invention, to improve the accuracy of sensing target obstacles in the environment, a combination of a sonar system and a lidar system is used to sense the target obstacles. The sonar system emits radio frequency pulses through multiple transducers. These pulses bounce in the environment, and the sonar system uses multiple microphones to receive the bounced acoustic reflection signals, obtaining acoustic reflection signal data collected by the sonar system for the target obstacle. Specifically, the sonar system uses a microphone array consisting of 16 microphones, which has a built-in digital-to-analog converter using pulse density modulation, eliminating the need for additional amplification of the signals collected by the microphone array. Optionally, the lidar system emits pulsed laser signals into the environment and receives the light reflection signals reflected by the target obstacle, thus enabling the sonar system and lidar system to collect acoustic and light reflection signal data of the target obstacle.

[0026] It should be noted that the acquisition of acoustic reflection signal data and optical reflection signal data of target obstacles by the sonar system and lidar system is continuous, and the processing of acoustic reflection signal data and optical reflection signal data in this embodiment of the invention is also continuous.

[0027] S102, determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data, and determine the fault status of the sonar system and the lidar system based on the distance fluctuation information.

[0028] In this embodiment of the invention, the distance fluctuation information of the target obstacle refers to the fluctuation of multiple distance result data obtained by the radar system in measuring the distance of the target obstacle over a continuous period of time. Generally, after receiving the acoustic reflection signal data of the target obstacle, the sonar system can determine the distance of the target obstacle from the sonar system based on the time difference between receiving the acoustic reflection signal and emitting the pulse. Similarly, the lidar can also determine the distance of the target obstacle based on the time difference between receiving the light reflection signal and emitting the laser signal. For the distance information of the same target obstacle collected by the lidar and sonar systems within a time period, the distance fluctuation information of the target obstacle collected by the lidar system and the sonar system can be determined, and whether the sonar system and the lidar system are malfunctioning can be determined based on the distance fluctuation information.

[0029] Specifically, for a sonar system, the distance to the same target obstacle collected should not fluctuate significantly within a short, continuous time period. When the distance to the same target obstacle collected by the sonar system fluctuates significantly within a short, continuous time period, it indicates that the sonar system is not accurately collecting the distance to the target obstacle, and this can be considered a malfunction of the sonar system. Similarly, when the distance to the same target obstacle collected by a lidar system fluctuates significantly within a short, continuous time period, it indicates that the lidar system is malfunctioning.

[0030] S103, determine the reference value and correction value for acoustic-optical radar data fusion based on the fault conditions of the sonar system and lidar system, and fuse the reference value and correction value to obtain acoustic-optical radar fused data.

[0031] In this embodiment of the invention, after determining the fault conditions of the sonar system and the lidar system, a reference value and a correction value for acoustic-optical radar data fusion are determined based on the fault information of the sonar system and the lidar system. The reference value and the correction value are then fused to obtain acoustic-optical radar fused data. The specific methods for determining the reference value and the correction value and the specific fusion method will be described in detail later in this invention.

[0032] The acoustic-optical radar data fusion method provided by this invention collects acoustic and optical reflection signal data of target obstacles using a sonar system and a lidar system. Based on the collected acoustic and optical reflection signal data, it determines the distance fluctuation information of the target obstacle acquired by the sonar and lidar systems. This distance fluctuation information is used to determine whether the sonar and lidar systems are malfunctioning. Based on the malfunction status of the sonar and lidar systems, it determines the baseline and correction values ​​for acoustic-optical radar data fusion, and then fuses these values ​​to obtain fused acoustic-optical radar data. Simultaneously using both lidar and sonar systems for environmental perception effectively avoids the limitations of a single radar system. Furthermore, by analyzing the distance fluctuation information of the target obstacle acquired by the sonar and lidar systems, it determines whether either system is malfunctioning. If one radar system malfunctions, the other radar system can be used to correct the detected target obstacle information, ensuring the accuracy of environmental perception and adapting to complex environmental requirements.

[0033] In some embodiments of the present invention, such as Figure 2 As shown, the acquisition of acoustic reflection signal data collected by the sonar system against the target obstacle includes: S201, acquire multi-angle acoustic reflection signal data collected by the sonar system for the target obstacle, and determine the time delay of the acoustic reflection signal data at each angle.

[0034] In this embodiment of the invention, the acoustic reflection signals of the target obstacle are collected by setting up multiple microphones. Since each microphone is set at a different angle and position, there will be a time delay when collecting the acoustic reflection signal data. Therefore, it is necessary to eliminate the time delay in the acoustic reflection signal data collected by each microphone. Since the relative position of each microphone channel to other microphone channels is known, the relative delay (relative distance / speed of sound) to the reference microphone (with zero delay) and other microphones can be measured in advance and created into a table. During calculation, the data can be directly looked up in the table and substituted into the table, which can effectively improve calculation efficiency and ensure real-time performance.

[0035] S202, a time-delay summation beamforming algorithm is used to fuse the acoustic reflection signal data at each angle based on the time delay of the acoustic reflection signal at each angle, so as to obtain the acoustic reflection signal data collected by the sonar system for the target obstacle.

[0036] In this embodiment of the invention, in order to enhance the intensity of the acoustic reflection signal data and compensate for the effects caused by the different arrival times due to the microphone's position, a delay-summing beamforming algorithm can be used to process the acoustic reflection signal. The expression for beamforming is:

[0037] in, The beam of the reflected sound signal For the first i The weight of each microphone, For the first i The directional angle of each microphone and pitch angle The joint function represents the first... i The azimuth information of each microphone is used to arrange the beams of each microphone into a matrix. In the matrix, rows represent different pitch angles and columns represent different azimuth angles. This allows us to represent the intensity of sound reflection signals from all directions, forming a spatial sound energy intensity matrix as follows:

[0038] in, This represents the distance-energy set of the target obstacle collected by k microphones.

[0039] The embodiments of the present invention can eliminate the time delay of the sound reflection signal by preprocessing the sound reflection signal and fix the maximum gain in one direction, thereby improving the accuracy of the sound reflection signal.

[0040] In some embodiments of the present invention, such as Figure 3 As shown, the distance fluctuation information of the target obstacle collected by the sonar system and lidar system is determined based on the acoustic reflection signal data and optical reflection signal data. The fault status of the sonar system and lidar system is then determined based on the distance fluctuation information, including: S301, using a preset depth belief network to extract the first distance fluctuation features of the target obstacle from the acoustic reflection signal data and the second distance fluctuation features of the target obstacle from the light reflection signal data.

[0041] In this embodiment of the invention, the malfunction of the sonar system and lidar system can be determined by collecting acoustic reflection signal data and optical reflection signal data. Specifically, this embodiment of the invention uses a deep belief network composed of stacked restricted Boltzmann machines to extract features and increase the dimensionality of the acoustic reflection signal data and optical reflection signal data. The structure of the deep belief network using stacked restricted Boltzmann machines in this embodiment of the invention is as follows: Figure 4 As shown, the deep belief network includes a coated restricted Boltzmann machine (RBM), and the restricted Boltzmann machine includes a data input layer. to and hidden layers to The input layer processes the raw data, while the hidden layer learns higher-order features from the input data. The hidden layer primarily uses weights and bias parameters to perform non-linear transformations on the data in the visible layer, extracting abstract features. The output layer... This represents the output of the deep belief network. The weights of the Nth layer Restricted Boltzmann Machine (RBM) are used to mine deep features from low-dimensional inputs, enhancing data representation capabilities. Specifically, by using acoustic reflection signal data and optical reflection signal data as inputs to the deep belief network (DBN) for two identification processes, the DBN can continuously extract feature relationships from these information and map them to higher dimensions, thereby extracting the first distance fluctuation features of the target obstacle from the acoustic reflection signal data and the second distance fluctuation features of the target obstacle from the optical reflection signal data.

[0042] S302 uses a pre-defined single-class support vector machine to classify the first and second distance fluctuation features of the target obstacle, and determines the fault status of the sonar system and lidar system.

[0043] In this embodiment of the invention, the extracted first and second range fluctuation features are input into a preset single-class support vector machine for classification, thereby determining the fault status of the sonar system and the lidar system. The structure of the single-class support vector machine is as follows: Figure 5 As shown, kernel function for:

[0044] in, and For the sample i and samples j Features These are preset parameters. to For Lagrange multipliers, For the first n Labels for each sample.

[0045] Specifically, when the distance to the same target obstacle collected by the sonar system fluctuates significantly within a short, continuous time period, it indicates that the sonar system's distance acquisition is inaccurate, and the sonar system is considered to be malfunctioning. Similarly, when the distance to the same target obstacle collected by the lidar system fluctuates significantly within a short, continuous time period, it indicates that the lidar system is malfunctioning.

[0046] This invention uses deep belief networks and single-class support vector machines to identify acoustic reflection signal data and optical reflection signal data, thereby improving the efficiency and accuracy of fault detection in sonar and lidar systems.

[0047] In some embodiments of the present invention, determining the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar system and the lidar system includes: When the sonar system malfunctions but the lidar system does not, the light reflection signal data is used as the reference value and the acoustic reflection signal data is used as the correction value. When the lidar system malfunctions but the sonar system does not, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value.

[0048] In this embodiment of the invention, due to the complexity of the radar system's detection environment, the sonar system or lidar system may become unusable or the detection data may be inaccurate. In this case, it is necessary to determine a reference value in both the lidar system and the sonar system, and to correct the detection data of the sonar system and lidar system based on this reference value. Specifically, when the sonar system malfunctions but the lidar system does not malfunction, the light reflection signal data is used as the reference value and the sound reflection signal data is used as the correction value; when the lidar system malfunctions but the sonar system does not malfunction, the sound reflection signal data is used as the reference value and the light reflection signal data is used as the correction value.

[0049] In some embodiments of the present invention, determining the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar system and the lidar system includes: When neither the sonar system nor the lidar system malfunctions, calculate the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data. When the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value. When the first variance is greater than the second variance, the light reflection signal data is used as the reference value, and the sound reflection signal data is used as the correction value.

[0050] In this embodiment of the invention, when neither the sonar system nor the lidar system malfunctions, a reference value and a correction value need to be determined based on the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data. Specifically, when the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value; when the first variance is greater than the second variance, the optical reflection signal data is used as the reference value and the acoustic reflection signal data is used as the correction value.

[0051] The embodiments of the present invention can ensure the accuracy of acoustic-optical radar data fusion by determining reference values ​​and correction values ​​in acoustic reflection signal data and optical reflection signal data based on the fault conditions of the sonar system and lidar system.

[0052] In some embodiments of the present invention, such as Figure 6 As shown, the baseline and correction values ​​are fused to obtain acoustic-optical radar fused data, including: S601 calculates the absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data.

[0053] In this embodiment of the invention, before fusing the reference value and the correction value, it is necessary to first determine the absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data to determine the difference in detection accuracy between the sonar system and the lidar system, and then determine the fusion parameters based on the difference in detection accuracy. Specifically, the average value of multiple acoustic reflection signal data and the average value of multiple optical reflection signal data can be calculated, and then the absolute value of the difference between the average value of the multiple acoustic reflection signal data and the average value of the multiple optical reflection signal data can be calculated.

[0054] S602 adjusts the fusion parameters in real time based on the relationship between the absolute value of the difference and the preset difference threshold.

[0055] In this embodiment of the invention, the selection of the fusion parameter is related to the magnitude of the absolute value of the difference and the preset difference threshold. Specifically, the adjustment formula for the fusion parameter is as follows:

[0056] in, and For a pre-set threshold, and , and These are the pre-set basic fusion parameters, and ; when When the detection difference between the sonar system and the lidar system is very small, it is considered a stable state of difference between the two radar systems; when When the detection difference between the sonar system and the lidar system is within a tolerable range, it is considered a relatively stable region of difference between the two radar systems; when At this point, the differences between the sonar system and the lidar system have exceeded the tolerable range, and this is considered a stage of significant differences between the two radar systems. and These are the pre-set basic fusion parameters. and The value of is determined by the initial state of the acoustic-optical data during data fusion.

[0057] S603 uses an adaptive Kalman filter algorithm to fuse the baseline and correction values ​​based on real-time adjusted fusion parameters to obtain fused acoustic-optical radar data.

[0058] In this embodiment of the invention, each time acoustic reflection signal data and optical reflection signal data are collected, they need to be fused. Specifically, an adaptive Kalman filter algorithm can be used to fuse the reference value and the correction value based on real-time adjusted fusion parameters. The fusion formula for the reference value and the correction value is as follows:

[0059] in, F For audio-visual fusion data, As the baseline value, This is a correction value.

[0060] As described in the previous embodiments, the fusion strategy for the reference value and the correction value falls into three categories. When both the lidar system and the sonar system are operating normally, the data from the original dataset with the smaller variance is used as the reference value, and the data from the dataset with the larger variance is used as the correction value for fusion. When either the lidar system or the sonar system determines that one of them has malfunctioned, data fusion can still be performed. In this case, the data from the normally operating lidar system is used as the reference value, and the data from the lidar system that is considered to have malfunctioned is used as the correction value for fusion. When both the lidar system and the sonar system determine that they have malfunctioned, an error occurs in the program, and the machine stops operating.

[0061] The embodiments of the present invention employ an adaptive Kalman filter algorithm, which can dynamically adjust the fusion parameters according to real-time environmental changes, thereby achieving accurate fusion of lidar and sonar data.

[0062] To better implement the acoustic-optical radar data fusion method in the embodiments of the present invention, based on the acoustic-optical radar data fusion method, correspondingly, as follows: Figure 7 As shown, this embodiment of the invention also provides an acoustic-optical radar data fusion device, the acoustic-optical radar data fusion device 700 comprising: The data acquisition module 701 is used to acquire acoustic reflection signal data and optical reflection signal data collected by the sonar system and lidar system against the target obstacle; The fault diagnosis module 702 is used to determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data, and to determine the fault status of the sonar system and the lidar system based on the distance fluctuation information. The data fusion module 703 is used to determine the reference value and correction value for acoustic-optical radar data fusion based on the fault conditions of the sonar system and lidar system, and to fuse the reference value and correction value to obtain acoustic-optical radar fused data.

[0063] The acoustic-optical radar data fusion device 700 provided in the above embodiments can realize the technical solutions described in the above acoustic-optical radar data fusion method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above acoustic-optical radar data fusion method embodiments, and will not be repeated here.

[0064] The acoustic-optical radar data fusion device provided in this invention can detect and correct faulty radar systems, and simultaneously acquire high-precision perception data of both long-range and short-range environments, thereby providing more comprehensive and accurate perception information in various complex environments. Due to the complementarity of sonar and lidar, they can work effectively under different environmental conditions, thus significantly improving the environmental adaptability and operational stability of the overall system.

[0065] like Figure 8 As shown, the present invention also provides an acoustic-optical fusion radar 800. The acoustic-optical fusion radar 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the acoustic-optical fusion radar 800 are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented instead.

[0066] In some embodiments, processor 801 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as the acoustic-optical radar data fusion method of the present invention.

[0067] In some embodiments, processor 801 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 801 may be local or remote. In some embodiments, processor 801 may be implemented on a cloud platform. In some embodiments, the cloud platform may include private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0068] In some embodiments, the memory 802 can be an internal storage unit of the acoustic-optical fusion radar 800, such as a hard disk or memory of the acoustic-optical fusion radar 800. In other embodiments, the memory 802 can also be an external storage device of the acoustic-optical fusion radar 800, such as a pluggable hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the acoustic-optical fusion radar 800.

[0069] Furthermore, the memory 802 may include both internal storage units of the acoustic-optical fusion radar 800 and external storage devices. The memory 802 is used to store the application software and various types of data installed on the acoustic-optical fusion radar 800.

[0070] In some embodiments, display 803 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 803 is used to display information from the acoustic-optical fusion radar 800 and to display a visual user interface. Components 801-803 of the acoustic-optical fusion radar 800 communicate with each other via a system bus.

[0071] In some embodiments, when the processor 801 executes the acoustic-optical-radar data fusion program in the memory 802, the following steps may be implemented: Acquire acoustic and optical reflection signal data collected by sonar and lidar systems against target obstacles; Based on acoustic reflection signal data and optical reflection signal data, determine the distance fluctuation information of the target obstacle collected by the sonar system and lidar system, and determine the fault status of the sonar system and lidar system based on the distance fluctuation information. The baseline and correction values ​​for acoustic-optical radar data fusion are determined based on the fault conditions of the sonar and lidar systems, and the baseline and correction values ​​are fused to obtain the acoustic-optical radar fused data.

[0072] It should be understood that when the processor 801 executes the acoustic-optical-radar data fusion program in the memory 802, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0073] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the acoustic-optical-radar data fusion methods provided in the above-described method embodiments.

[0074] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program of the methods described in the above embodiments can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0075] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fusion of acoustic and optical radar data, characterized in that, include: Acquire acoustic and optical reflection signal data collected by sonar and lidar systems against target obstacles; Based on the acoustic reflection signal data and the optical reflection signal data, the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system is determined, and the fault status of the sonar system and the lidar system is determined based on the distance fluctuation information; Based on the fault conditions of the sonar system and the lidar system, the baseline value and correction value for acoustic-optical radar data fusion are determined, and the baseline value and the correction value are fused to obtain acoustic-optical radar fused data. The process of fusing the baseline value and the correction value to obtain acoustic-optical radar fused data includes: Calculate the absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data; The fusion parameters are adjusted in real time based on the relationship between the absolute value of the difference and the preset difference threshold. An adaptive Kalman filter algorithm is used to fuse the baseline value and the correction value based on the real-time adjusted fusion parameters to obtain acoustic-optical radar fused data. The formula for adjusting the fusion parameters is: in, The absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data. and For a pre-set threshold, and , and These are the pre-set basic fusion parameters, and ; The formula for fusing the baseline value and the correction value is as follows: in, F For audio-visual fusion data, As the baseline value, This is a correction value.

2. The acoustic-optical radar data fusion method according to claim 1, characterized in that, Acquire acoustic reflection signal data collected by the sonar system against the target obstacle, including: Acquire multi-angle acoustic reflection signal data collected by the sonar system against the target obstacle, and determine the time delay of the acoustic reflection signal data at each angle; A time-delay summation beamforming algorithm is used to fuse acoustic reflection signal data from each angle based on the time delay of the acoustic reflection signal from each angle, so as to obtain acoustic reflection signal data collected by the sonar system for the target obstacle.

3. The acoustic-optical radar data fusion method according to claim 1, characterized in that, The step of determining the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data, and determining the fault status of the sonar system and the lidar system based on the distance fluctuation information, includes: A preset deep belief network is used to extract the first distance fluctuation features of the target obstacle from the acoustic reflection signal data and the second distance fluctuation features of the target obstacle from the optical reflection signal data; A preset single-class support vector machine is used to classify the first distance fluctuation feature and the second distance fluctuation feature of the target obstacle to determine the fault status of the sonar system and the lidar system.

4. The acoustic-optical radar data fusion method according to claim 1, characterized in that, The determination of the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar system and the lidar system includes: When the sonar system malfunctions but the lidar system does not malfunction, the light reflection signal data is used as the reference value and the acoustic reflection signal data is used as the correction value. When the lidar system malfunctions but the sonar system does not malfunction, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value.

5. The acoustic-optical radar data fusion method according to claim 1, characterized in that, The determination of the baseline and correction values ​​for acoustic-optical radar data fusion based on the fault conditions of the sonar system and the lidar system includes: When neither the sonar system nor the lidar system malfunctions, calculate the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data; When the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value. When the first variance is greater than the second variance, the light reflection signal data is used as the reference value, and the sound reflection signal data is used as the correction value.

6. An acoustic-optical radar data fusion device, applicable to the acoustic-optical radar data fusion method according to any one of claims 1 to 5, characterized in that, include: The data acquisition module is used to acquire acoustic reflection signal data and optical reflection signal data collected by the sonar system and lidar system against the target obstacle; The fault diagnosis module is used to determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data, and to determine the fault status of the sonar system and the lidar system based on the distance fluctuation information; The data fusion module is used to determine the baseline value and correction value for acoustic-optical radar data fusion based on the fault conditions of the sonar system and the lidar system, and to fuse the baseline value and the correction value to obtain acoustic-optical radar fused data.

7. An acoustic-optical fusion radar, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the acoustic-optical-radar data fusion method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the acoustic-optical radar data fusion method according to any one of claims 1 to 5.

Images