Acousto-optic radar data fusion method and device, radar and computer storage medium

Abstract

The invention relates to an acousto-optic radar data fusion method and device, a radar and a computer storage medium, and belongs to the technical field of radars, and the acousto-optic radar data fusion method comprises the steps: obtaining sound reflection signal data and light reflection signal data collected by a sonar system and a laser radar system for a target obstacle; determining distance fluctuation information of a target obstacle collected by a sonar system and a laser radar system according to the sound reflection signal data and the light reflection signal data, and determining fault conditions of the sonar system and the laser radar system according to the distance fluctuation information; and determining a reference value and a correction value of acousto-optic radar data fusion according to the fault conditions of the sonar system and the laser radar system, and fusing the reference value and the correction value to obtain acousto-optic radar fusion data. The method can guarantee the accuracy of environment perception, and is suitable for complex environment requirements.

Description

Technical Field

[0001] The present invention relates to the field of radar technology, and in particular, to an acoustic-optic radar data fusion method, device, radar, and computer storage medium. Background Art

[0002] With the development of technologies such as intelligent perception and driverless driving, the requirements for environmental perception technology are getting higher and higher.

[0003] In the field of modern intelligent perception and navigation technology, the development of radar technology has greatly promoted the progress of robot and autonomous driving technologies. Traditional radars include lidar and sonar systems. The lidar system has good single precision in long-distance environmental perception, but in some environments, affected by many factors, the detection accuracy drops significantly or even fails completely, such as complex environments like smoke-filled, underwater, and high-reflectivity surfaces; while the sonar system, although inferior to lidar in terms of accuracy and resolution, shows stronger detection capabilities in close-range and complex environments, such as scenes like smoke-filled, dark, or underwater.

[0004] It can be seen that a single radar detection system cannot meet complex modern requirements and cannot achieve accurate perception of the entire environment. Summary of the Invention

[0005] In view of this, it is necessary to provide an acoustic-optic radar data fusion method, device, radar, and computer storage medium to solve the problem that a single radar detection system cannot meet complex modern requirements and cannot achieve accurate perception of the entire environment.

[0006] To solve the above problems, in a first aspect, the present invention provides an acoustic-optic radar data fusion method, including: Obtaining acoustic reflection signal data and optical reflection signal data collected by the sonar system and the lidar system for a target obstacle; Determining the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system according to the acoustic reflection signal data and the optical reflection signal data, and determining the fault conditions of the sonar system and the lidar system according to the distance fluctuation information; Determining the reference value and correction value for acoustic-optic radar data fusion according to the fault conditions of the sonar system and the lidar system, and fusing the reference value and the correction value to obtain the acoustic-optic radar fusion data.

[0007] In a possible implementation manner, obtaining the acoustic reflection signal data collected by the sonar system for a target obstacle includes: Obtaining multi-angle acoustic reflection signal data collected by the sonar system for a target obstacle, and determining the time delay duration of each angle of acoustic reflection signal data; The delay-sum beamforming algorithm is used to fuse the acoustic reflection signal data at each angle based on the time delay duration of the acoustic reflection signals at each angle, and the acoustic reflection signal data collected by the sonar system for the target obstacle is obtained.

[0008] In a possible implementation, the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system is determined according to the acoustic reflection signal data and the optical reflection signal data, and the fault conditions of the sonar system and the lidar system are determined based on the distance fluctuation information, including: The first distance fluctuation feature of the target obstacle of the acoustic reflection signal data and the second distance fluctuation feature of the target obstacle of the optical reflection signal data are extracted by using a preset deep belief network; The first distance fluctuation feature and the second distance fluctuation feature of the target obstacle are classified by using a preset one-class support vector machine to determine the fault conditions of the sonar system and the lidar system.

[0009] In a possible implementation, the reference value and the correction value for the fusion of the acoustic-optic radar data are determined according to the fault conditions of the sonar system and the lidar system, including: When the sonar system fails and the lidar system does not fail, the optical reflection signal data is used as the reference value and the acoustic reflection signal data is used as the correction value; When the lidar system fails and the sonar system does not fail, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value.

[0010] In a possible implementation, the reference value and the correction value for the fusion of the acoustic-optic radar data are determined according to the fault conditions of the sonar system and the lidar system, including: When neither the sonar system nor the lidar system fails, the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data are calculated; When the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value and the optical reflection signal data is used as the correction value; When the first variance is greater than the second variance, the optical reflection signal data is used as the reference value and the acoustic reflection signal data is used as the correction value.

[0011] In a possible implementation, the reference value and the correction value are fused to obtain the acoustic-optic radar fusion data, including: The absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data is calculated; The fusion parameter is adjusted in real time according to the magnitude relationship between the absolute value of the difference and a preset difference threshold; The reference value and the correction value are fused by using an adaptive Kalman filtering algorithm based on the fusion parameter adjusted in real time to obtain the acoustic-optic radar fusion data.

[0012] In a possible implementation, the adjustment formula for the fusion parameter is as follows:

[0013] Wherein, and are preset thresholds, and , and are preset basic fusion parameters, and ; The fusion formula for the reference value and the correction value is as follows:

[0014] Wherein, F is the acoustic-optic fusion data, is the reference value, is the correction value.

[0015] In a second aspect, the present invention further provides an acoustic-optic radar data fusion device, which is characterized by comprising: A data acquisition module, configured to acquire acoustic reflection signal data and optical reflection signal data collected by a sonar system and a lidar system for a target obstacle; A fault judgment module, configured to determine distance fluctuation information of the target obstacle collected by the sonar system and the lidar system according to the acoustic reflection signal data and the optical reflection signal data, and determine the fault conditions of the sonar system and the lidar system according to the distance fluctuation information; A data fusion module, configured to determine a reference value and a correction value for acoustic-optic radar data fusion according to the fault conditions of the sonar system and the lidar system, and fuse the reference value and the correction value to obtain acoustic-optic radar fusion data.

[0016] In a third aspect, the present invention further provides an acoustic-optic fusion radar, which is characterized by comprising a memory and a processor, wherein, The memory is configured to store a program; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps in the acoustic-optic radar data fusion method described in any one of the above embodiments.

[0017] In a fourth aspect, the present invention further provides a computer-readable storage medium, which is characterized by being configured to store a computer-readable program or instruction, and when the program or instruction is executed by a processor, it can implement the steps in the acoustic-optic radar data fusion method described in any one of the above embodiments.

[0018] The beneficial effects of the present invention are as follows: The method for fusing acoustic and optical radar data provided by the present invention collects acoustic reflection signal data and optical reflection signal data of a target obstacle through a sonar system and a lidar system, determines the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system according to the collected acoustic reflection signal data and optical reflection signal data, determines whether the sonar system and the lidar system are faulty according to the distance fluctuation information, determines the reference value and the correction value for fusing the acoustic and optical radar data according to the fault conditions of the sonar system and the lidar system, and fuses the reference value and the correction value to obtain the fused acoustic and optical radar data. At the same time, the lidar system and the sonar system are used to perceive the environment, which can effectively avoid the limitations of a single radar system in environmental perception. At the same time, by analyzing the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system, it is determined whether the sonar system and the lidar system are faulty. When one of the radar systems fails, the other radar system can be used to correct the detected target obstacle information to ensure the accuracy of environmental perception and meet the requirements of complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic flow chart of a method for fusing acoustic and optical radar data provided by an embodiment of the present invention; Figure 2 It is a schematic flow chart of a method for obtaining acoustic reflection signal data provided by an embodiment of the present invention; Figure 3 It is a schematic flow chart of a fault judgment method provided by an embodiment of the present invention; Figure 4 It is a structural diagram of a deep belief network stacked with restricted Boltzmann machines provided by an embodiment of the present invention; Figure 5 It is a structural diagram of a one-class support vector machine provided by an embodiment of the present invention; Figure 6 It is a schematic flow chart of a data fusion method provided by an embodiment of the present invention; Figure 7 It is a schematic structural diagram of an acoustic and optical radar data fusion device provided by an embodiment of the present invention; Figure 8 It is a schematic structural diagram of an acoustic and optical fusion radar provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The preferred embodiments of the present invention will be specifically described below with reference to the accompanying drawings. The accompanying drawings form a part of this application and are used together with the embodiments of the present invention to explain the principles of the present invention, rather than to limit the scope of the present invention.

[0022] In the embodiments of the present invention, the descriptions such as "first", "second", etc. are only for descriptive purposes and cannot be construed as indicating or implying their relative importance or implicitly specifying the quantity of the indicated technical features. Therefore, the technical features defined with "first" and "second" may explicitly or implicitly include at least one such feature.

[0023] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0024] A specific embodiment of the present invention, as Figure 1 shown, discloses an acoustic-optic radar data fusion method, including: S101, obtaining acoustic reflection signal data and optical reflection signal data collected by a sonar system and a lidar system for a target obstacle.

[0025] In the embodiments of the present invention, in order to improve the perception accuracy of the target obstacle in the environment, a combination of a sonar system and a lidar system is used to perceive the target obstacle. Among them, the sonar system emits frequency band pulses through multiple transducers, and these pulses bounce back in the environment. The sonar system uses multiple microphones to receive the bounced acoustic reflection signals, and obtains the acoustic reflection signal data collected by the sonar system for the target obstacle. Specifically, the sonar system uses a microphone array composed of 16 microphones, which is built-in with a digital-to-analog converter using pulse density modulation, which makes the signals collected by the microphone array no longer need additional amplification. Optionally, the lidar system emits pulsed laser signals to the environment and receives the optical reflection signals reflected by the target obstacle, so as to realize the acquisition of the acoustic reflection signal data and the optical reflection signal data of the target obstacle by the sonar system and the lidar system.

[0026] It should be noted that the acquisition of the acoustic reflection signal data and the optical reflection signal data of the target obstacle by the sonar system and the lidar system is continuous, and the processing of the acoustic reflection signal data and the optical reflection signal data in the embodiments of the present invention is also continuous.

[0027] S102. Determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system based on the acoustic reflection signal data and the optical reflection signal data, and determine the fault conditions of the sonar system and the lidar system according to the distance fluctuation information.

[0028] In the embodiment of the present invention, the distance fluctuation information of the target obstacle refers to the fluctuation of multiple distance result data obtained by the radar system when ranging the target obstacle continuously for a period of time. Generally, after the sonar system receives the acoustic reflection signal data of the target obstacle, it can determine the distance between the target obstacle and the sonar system according to the time difference between the received acoustic reflection signal and the emitted pulse. Similarly, the lidar can also determine the distance of the target obstacle according to the time difference between the received optical reflection signal and the emitted laser signal. For the distance information of the same target obstacle collected by the lidar and the sonar system within a period of time, the distance fluctuation information of the target obstacle collected by the lidar system and the sonar system can be determined, and whether the sonar system and the lidar system are faulty can be determined according to the distance fluctuation information.

[0029] Specifically, for the sonar system, the distance of the same target obstacle collected will not have large fluctuations within a short continuous time period. When the distance of the same target obstacle collected by the sonar system has large fluctuations within a short continuous time period, it means that the sonar system's acquisition of the target obstacle's distance is inaccurate. At this time, it can be determined that the sonar system has a fault. Similarly, when the distance of the same target obstacle collected by the lidar system has large fluctuations within a short continuous time period, it means that the lidar system has a fault.

[0030] S103. Determine the reference value and correction value for the acoustic-optical radar data fusion according to the fault conditions of the sonar system and the lidar system, and fuse the reference value and the correction value to obtain the acoustic-optical radar fusion data.

[0031] In the embodiment of the present invention, after determining the fault conditions of the sonar system and the lidar system, the reference value and correction value for the acoustic-optical radar data fusion are determined according to the fault information of the sonar system and the lidar system, and the reference value and the correction value are fused to obtain the acoustic-optical radar fusion data. The specific determination method and specific fusion method for the reference value and the correction value will be described in detail later in the present invention.

[0032] The acoustic-optic radar data fusion method provided by the present invention collects acoustic reflection signal data and optical reflection signal data of a target obstacle through a sonar system and a lidar system, determines the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system according to the collected acoustic reflection signal data and optical reflection signal data, determines whether the sonar system and the lidar system are faulty according to the distance fluctuation information, determines the reference value and the correction value of the acoustic-optic radar data fusion according to the fault conditions of the sonar system and the lidar system, and fuses the reference value and the correction value to obtain the acoustic-optic radar fusion data. At the same time, the lidar system and the sonar system are used to perceive the environment, which can effectively avoid the limitations of a single radar system for environmental perception. At the same time, by analyzing the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system, it is determined whether the sonar system and the lidar system are faulty. When one of the radar systems fails, the other radar system can be used to correct the detected target obstacle information to ensure the accuracy of environmental perception and meet the requirements of complex environments.

[0033] In some embodiments of the present invention, as Figure 2 shown, obtaining the acoustic reflection signal data collected by the sonar system for the target obstacle includes: S201, obtaining multi-angle acoustic reflection signal data collected by the sonar system for the target obstacle, and determining the time delay duration of each angle of acoustic reflection signal data.

[0034] In the embodiments of the present invention, for the acoustic reflection signal of the target obstacle, multiple microphones are set for collection. Since the angles and positions of each microphone are different, there will be a time delay when collecting the acoustic reflection signal data, and it is necessary to eliminate the time delay of the acoustic reflection signal data collected by each microphone. Since the relative positions of each microphone channel and other microphone channels are known, the relative delays (relative distances / sound speed) from the reference microphone (delay is 0) to other microphones can be measured in advance and made into a table, and directly look up the table and substitute it into the calculation during calculation, which can effectively improve the calculation efficiency and ensure real-time performance.

[0035] S202, using the delay-and-sum beamforming algorithm to fuse the acoustic reflection signal data of each angle based on the time delay duration of the acoustic reflection signals of each angle, and obtaining the acoustic reflection signal data collected by the sonar system for the target obstacle.

[0036] In the embodiments of the present invention, in order to strengthen the intensity of the acoustic reflection signal data and compensate for the influence caused by the different arrival times due to the position relationship of the microphones, the delay-and-sum beamforming algorithm can be used to process the acoustic reflection signal. Among them, the expression of beamforming is:

[0037] Among them, is the beam of the acoustic reflection signal, is the weight of the i th microphone, is the i th microphone's azimuth angle and elevation angle joint function, representing the azimuth information of the i th microphone. Arrange the beams of each microphone in a matrix. The rows in the matrix represent different elevation angles, and the columns represent different azimuth angles. The data intensity of the acoustic reflection signals in all directions around can be represented, forming a matrix of spatial acoustic energy intensity as follows:

[0038] Among them, represents the distance - energy set of the target obstacle collected by k microphones.

[0039] In the embodiment of the present invention, by pre - processing the acoustic reflection signal, the time delay of the acoustic reflection signal can be eliminated, and the maximum gain can be fixed in one direction, improving the accuracy of the acoustic reflection signal.

[0040] In some embodiments of the present invention, as Figure 3 shown, according to the acoustic reflection signal data and the optical reflection signal data, the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system is determined, and according to the distance fluctuation information, the fault conditions of the sonar system and the lidar system are determined, including: S301, using a preset deep belief network to extract the first distance fluctuation feature of the target obstacle from the acoustic reflection signal data and the second distance fluctuation feature of the target obstacle from the optical reflection signal data.

[0041] In the embodiment of the present invention, it is possible to determine whether the sonar system and the lidar system are faulty through the collected acoustic reflection signal data and optical reflection signal data. Specifically, in the embodiment of the present invention, a deep belief network stacked by restricted Boltzmann machines is used to extract features and perform dimensionality elevation on the acoustic reflection signal data and the optical reflection signal data. The structure of the deep belief network stacked by restricted Boltzmann machines adopted in the embodiment of the present invention is as Figure 4 shown, among which, the deep belief network includes a stacked restricted Boltzmann machine (RBM). The restricted Boltzmann machine includes a data input layer to and a hidden layer to . The data input layer is used to process the original data, and the hidden layer is used to learn the high - order features of the input data. The hidden layer mainly performs a non - linear transformation on the data of the visible layer through weight and bias parameters to extract abstract features. The output layer Represents the output of the deep belief network. Is the weight of the Nth layer restricted Boltzmann machine. By using the restricted Boltzmann machine to mine deep features from low-dimensional inputs, the data representation ability is enhanced. Specifically, by taking the acoustic reflection signal data and the optical reflection signal data as the inputs of the deep belief network respectively and performing two identifications, the deep belief network can continuously extract the feature relationships in this information and map them to a higher dimension, thereby extracting the first distance fluctuation feature of the target obstacle in the acoustic reflection signal data and the second distance fluctuation feature of the target obstacle in the optical reflection signal data.

[0042] S302. Use a preset one-class support vector machine to classify the first distance fluctuation feature and the second distance fluctuation feature of the target obstacle, and determine the fault conditions of the sonar system and the lidar system.

[0043] In the embodiments of the present invention, by inputting the extracted first distance fluctuation feature and the second distance fluctuation feature into a preset one-class support vector machine for classification, the fault conditions of the sonar system and the lidar system can be determined. The structure of the one-class support vector machine is as Figure 5 shown, and the kernel function is:

[0044] Among them, and are the features of sample i and sample j , is a preset parameter. to are Lagrange multipliers, is the n th sample label.

[0045] Specifically, when there are large fluctuations in the distances of the same target obstacle collected by the sonar system within a short continuous time period, it indicates that the sonar system's acquisition of the target obstacle distance is inaccurate. At this time, it can be determined that the sonar system has a fault. Similarly, when there are large fluctuations in the distances of the same target obstacle collected by the lidar system within a short continuous time period, it indicates that the lidar system has a fault.

[0046] The embodiments of the present invention identify the acoustic reflection signal data and the optical reflection signal data through the deep belief network and the one-class support vector machine, improving the efficiency and accuracy of fault detection for the sonar system and the lidar system.

[0047] In some embodiments of the present invention, determining the reference value and correction value for the fusion of acoustic and optical radar data according to the fault conditions of the sonar system and the lidar system includes: When the sonar system fails and the lidar system does not fail, the optical reflection signal data is used as the reference value, and the acoustic reflection signal data is used as the correction value; When the lidar system fails and the sonar system does not fail, the acoustic reflection signal data is used as the reference value, and the optical reflection signal data is used as the correction value.

[0048] In the embodiments of the present invention, due to the complexity of the detection environment of the radar system, it may cause the sonar system or the lidar system to be unusable or the detection data to be inaccurate. At this time, a reference value needs to be determined between the lidar system and the sonar system, and the detection data of the sonar system and the lidar system is corrected based on this reference value. Specifically, when the sonar system fails and the lidar system does not fail, the optical reflection signal data is used as the reference value, and the acoustic reflection signal data is used as the correction value; when the lidar system fails and the sonar system does not fail, the acoustic reflection signal data is used as the reference value, and the optical reflection signal data is used as the correction value.

[0049] In some embodiments of the present invention, determining the reference value and the correction value for the fusion of the acoustic and optical radar data according to the failure conditions of the sonar system and the lidar system includes: When neither the sonar system nor the lidar system fails, calculate the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data; When the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value, and the optical reflection signal data is used as the correction value; When the first variance is greater than the second variance, the optical reflection signal data is used as the reference value, and the acoustic reflection signal data is used as the correction value.

[0050] In the embodiments of the present invention, when neither the sonar system nor the lidar system fails, it is necessary to determine the reference value and the correction value according to the first variance of the acoustic reflection signal data and the second variance of the optical reflection signal data. Specifically, when the first variance is less than or equal to the second variance, the acoustic reflection signal data is used as the reference value, and the optical reflection signal data is used as the correction value; when the first variance is greater than the second variance, the optical reflection signal data is used as the reference value, and the acoustic reflection signal data is used as the correction value.

[0051] The embodiments of the present invention can ensure the accuracy of the fusion of the acoustic and optical radar data by determining the reference value and the correction value from the acoustic reflection signal data and the optical reflection signal data according to the failure conditions of the sonar system and the lidar system.

[0052] In some embodiments of the present invention, as Figure 6 shown, fusing the reference value and the correction value to obtain the fused acoustic and optical radar data includes: S601 Calculate the absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data.

[0053] In the embodiment of the present invention, before fusing the reference value and the correction value, it is necessary to first determine the absolute value of the difference between the acoustic reflection signal data and the optical reflection signal data to determine the detection accuracy difference between the sonar system and the lidar system, and determine the fusion parameter according to the detection accuracy difference. Specifically, the average value of multiple acoustic reflection signal data and the average value of multiple optical reflection signal data can be calculated, and then the absolute value of the difference between the average value of multiple acoustic reflection signal data and the average value of multiple optical reflection signal data is calculated.

[0054] S602, adjust the fusion parameter in real time according to the magnitude relationship between the absolute value of the difference and the preset difference threshold.

[0055] In the embodiment of the present invention, the selection of the fusion parameter is related to the magnitude of the absolute value of the difference and the preset difference threshold. Specifically, the adjustment formula of the fusion parameter is:

[0056] Wherein, and are preset thresholds, and , and are preset basic fusion parameters, and ; When , it means that the detection difference between the sonar system and the lidar system is very small, regarded as the difference stable state of the two radar systems; when , it means that the detection difference between the sonar system and the lidar system is within the tolerable range, regarded as the relatively stable region of the difference between the two radar systems; when , it means that the difference between the sonar system and the lidar system has exceeded the tolerable range, regarded as the large difference stage of the two radar systems and are preset basic fusion parameters, and The values of are to determine the initial situation when the acoustic and optical data are fused.

[0057] S603, use the adaptive Kalman filter algorithm to fuse the reference value and the correction value based on the fusion parameter adjusted in real time to obtain the acoustic-optic radar fusion data.

[0058] In the embodiment of the present invention, for the acoustic reflection signal data and the optical reflection signal data collected each time, fusion is required. Specifically, the adaptive Kalman filter algorithm can be used to fuse the reference value and the correction value based on the fusion parameter adjusted in real time. The fusion formula of the reference value and the correction value is:

[0059] Among them, F is the acoustic-optic fusion data, is the reference value, is the correction value.

[0060] As described in the foregoing embodiments, the fusion strategy of the reference value and the correction value has three cases. When both the lidar system and the sonar system are working properly, at this time, the data with the smaller variance in the original data is used as the reference value, and the data with the larger variance is used as the correction value for fusion. When it is considered that one of the lidar system or the sonar system has a fault, at this time, it is considered that although the equipment of the faulty party has a fault, data fusion can still be performed. At this time, the data of the normally working radar system is used as the reference value, and the data of the radar system considered to have a fault is used as the correction value for fusion. When both the lidar system and the sonar system are considered to have a fault, then the program reports an error at this time and the machine stops running.

[0061] The embodiment of the present invention adopts an adaptive Kalman filtering algorithm, which can dynamically adjust the fusion parameters according to the real-time environmental changes, so as to achieve the precise fusion of lidar and sonar data.

[0062] In order to better implement the acoustic-optic radar data fusion method in the embodiment of the present invention, correspondingly, on the basis of the acoustic-optic radar data fusion method, as Figure 7 shown, the embodiment of the present invention also provides an acoustic-optic radar data fusion device. The acoustic-optic radar data fusion device 700 includes: A data acquisition module 701, configured to acquire acoustic reflection signal data and optical reflection signal data collected by the sonar system and the lidar system for a target obstacle; A fault judgment module 702, configured to determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system according to the acoustic reflection signal data and the optical reflection signal data, and determine the fault conditions of the sonar system and the lidar system according to the distance fluctuation information; A data fusion module 703, configured to determine the reference value and the correction value for the acoustic-optic radar data fusion according to the fault conditions of the sonar system and the lidar system, and fuse the reference value and the correction value to obtain the acoustic-optic radar fusion data.

[0063] The acoustic-optic radar data fusion device 700 provided in the foregoing embodiment can implement the technical solutions described in the embodiment of the acoustic-optic radar data fusion method. For the specific implementation principles of the foregoing modules or units, reference may be made to the corresponding content in the embodiment of the acoustic-optic radar data fusion method described above, which will not be elaborated here.

[0064] The acoustic-optic radar data fusion device provided by the embodiment of the present invention can detect and correct a faulty radar system, and can simultaneously obtain high-precision perception data of the long-distance and short-distance environments, so as to provide more comprehensive and accurate perception information in various complex environments. Due to the complementarity of sonar and lidar, it can work effectively under different environmental conditions, thus greatly improving the environmental adaptability and working stability of the overall system.

[0065] As Figure 8 shown, the present invention also correspondingly provides an acoustic-optic fusion radar 800. The acoustic-optic fusion radar 800 includes a processor 801, a memory 802, and a display 803. Figure 8 Only some components of the acoustic-optic fusion radar 800 are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0066] In some embodiments, the processor 801 may be a central processing unit (CPU), a microprocessor, or other data processing chips, and is used to run the program code stored in the memory 802 or process data, such as the acoustic-optic radar data fusion method in the present invention.

[0067] In some embodiments, the processor 801 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 801 may be local or remote. In some embodiments, the processor 801 may be implemented on a cloud platform. In some embodiments, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination of the above.

[0068] In some embodiments, the memory 802 may be an internal storage unit of the acoustic-optic fusion radar 800, such as the hard disk or memory of the acoustic-optic fusion radar 800. In some other embodiments, the memory 802 may also be an external storage device of the acoustic-optic fusion radar 800, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the acoustic-optic fusion radar 800.

[0069] Furthermore, the memory 802 may also include both the internal storage unit of the acoustic-optic fusion radar 800 and the external storage device. The memory 802 is used to store the application software installed on the acoustic-optic fusion radar 800 and various types of data.

[0070] The display 803 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. in some embodiments. The display 803 is used to display the information of the acoustic-optic fusion radar 800 and to display a visual user interface. The components 801-803 of the acoustic-optic fusion radar 800 communicate with each other through the system bus.

[0071] In some embodiments, when the processor 801 executes the acoustic-optic radar data fusion program in the memory 802, the following steps can be implemented: Obtain the acoustic reflection signal data and the optical reflection signal data collected by the sonar system and the lidar system for the target obstacle; Determine the distance fluctuation information of the target obstacle collected by the sonar system and the lidar system according to the acoustic reflection signal data and the optical reflection signal data, and determine the fault conditions of the sonar system and the lidar system according to the distance fluctuation information; Determine the reference value and the correction value of the acoustic-optic radar data fusion according to the fault conditions of the sonar system and the lidar system, and fuse the reference value and the correction value to obtain the acoustic-optic radar fusion data.

[0072] It should be understood that when the processor 801 executes the acoustic-optic radar data fusion program in the memory 802, in addition to the above functions, other functions can also be implemented. For specific details, please refer to the description of the corresponding method embodiments above.

[0073] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium is used to store a computer-readable program or instruction. When the program or instruction is executed by a processor, the steps or functions in the acoustic-optic radar data fusion method provided by the above method embodiments can be implemented.

[0074] Those skilled in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The programs of the methods in the above embodiments can be stored in a computer-readable storage medium. Among them, the computer-readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory, etc.

[0075] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for fusion of acoustic and optical radar data, characterized in that: include: Acquire the acoustic reflection signal data and optical reflection signal data collected by the sonar system and the lidar system for the target obstacle; Determine distance fluctuation information of the target obstacle collected by the sonar system and the laser radar system according to the acoustic reflection signal data and the light reflection signal data, and determine the fault conditions of the sonar system and the laser radar system according to the distance fluctuation information; A reference value and a correction value for the fusion of acoustic and optical radar data are determined according to the fault conditions of the sonar system and the laser radar system, and the reference value and the correction value are fused to obtain acoustic and optical radar fusion data.

2. The method for fusion of acoustic and optical radar data according to claim 1, characterized in that: Acquire the acoustic reflection signal data collected by the sonar system for the target obstacle, including: Obtain multi-angle acoustic reflection signal data collected by the sonar system for the target obstacle, and determine the time delay length of the acoustic reflection signal data at each angle; The delayed sum beamforming algorithm is used to fuse the acoustic reflection signal data at each angle based on the delay length of the acoustic reflection signal at each angle, and the acoustic reflection signal data collected by the sonar system for the target obstacle is obtained.

3. The method for fusion of acoustic and optical radar data according to claim 1, characterized in that: The determining of distance fluctuation information of the target obstacle collected by the sonar system and the laser radar system according to the acoustic reflection signal data and the light reflection signal data, and determining the fault conditions of the sonar system and the laser radar system according to the distance fluctuation information, comprises: A preset deep belief network is used to extract a first distance fluctuation feature of the target obstacle of the acoustic reflection signal data and a second distance fluctuation feature of the target obstacle of the light reflection signal data; A preset single-class support vector machine is used to classify the first distance fluctuation feature and the second distance fluctuation feature of the target obstacle to determine the fault conditions of the sonar system and the lidar system.

4. The method for fusion of acoustic and optical radar data according to claim 1, characterized in that: The step of determining the reference value and correction value of the acoustic and optical radar data fusion according to the fault conditions of the sonar system and the laser radar system includes: When the sonar system fails and the laser radar system does not fail, the light reflection signal data is used as a reference value and the sound reflection signal data is used as a correction value; When the laser radar system fails and the sonar system does not fail, the acoustic reflection signal data is used as a reference value and the light reflection signal data is used as a correction value.

5. The method for fusion of acoustic and optical radar data according to claim 1, characterized in that: The step of determining the reference value and correction value of the acoustic and optical radar data fusion according to the fault conditions of the sonar system and the laser radar system includes: When neither the sonar system nor the laser radar system fails, calculating a first variance of the acoustic reflection signal data and a second variance of the optical reflection signal data; When the first variance is less than or equal to the second variance, taking the acoustic reflection signal data as a reference value and taking the light reflection signal data as a correction value; When the first variance is greater than the second variance, the light reflection signal data is used as a reference value, and the sound reflection signal data is used as a correction value.

6. The method for fusion of acoustic and optical radar data according to claim 1, characterized in that: The step of fusing the reference value and the correction value to obtain acoustic and optical radar fusion data includes: Calculating the absolute value of the difference between the acoustic reflection signal data and the light reflection signal data; Adjusting the fusion parameter in real time according to the magnitude relationship between the absolute value of the difference and a preset difference threshold; The reference value and the correction value are fused based on the fusion parameter adjusted in real time by using an adaptive Kalman filter algorithm to obtain acoustic and optical radar fusion data.

7. The method for fusion of acoustic and optical radar data according to claim 6, characterized in that: The adjustment formula of the fusion parameter is: in, and is a preset threshold, and , and is the pre-set basic fusion parameter, and ; The fusion formula of the reference value and the correction value is: in, F For the sound and light fusion data, is the base value, is the correction value.

8. An acoustic and optical radar data fusion device, characterized in that: include: A data acquisition module is used to acquire acoustic reflection signal data and optical reflection signal data collected by the sonar system and the lidar system for target obstacles; a fault judgment module, used to determine the distance fluctuation information of the target obstacle collected by the sonar system and the laser radar system according to the acoustic reflection signal data and the light reflection signal data, and determine the fault conditions of the sonar system and the laser radar system according to the distance fluctuation information; The data fusion module is used to determine the reference value and correction value of the acoustic and optical radar data fusion according to the fault conditions of the sonar system and the lidar system, and fuse the reference value and the correction value to obtain the acoustic and optical radar fusion data.

9. An acoustic-optical fusion radar, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the acoustic and optical radar data fusion method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the acoustic and optical radar data fusion method described in any one of claims 1 to 7.

Images