Indoor rescue vehicle based on audio-visual combination and trapped person positioning method

By integrating audio-visual technology and lidar on indoor rescue vehicles, combined with neural networks and object detection algorithms, the problem of positioning difficulties in indoor rescue is solved, and the rapid and accurate positioning of trapped people and the improvement of rescue efficiency is achieved.

CN119084724APending Publication Date: 2024-12-06HARBIN UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411191146.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing indoor rescue equipment and trapped personnel positioning methods are difficult to quickly and accurately locate trapped personnel in complex indoor environments, resulting in an extended rescue time.

Method used

Using an indoor rescue vehicle based on audio-visual combination, equipped with a binocular depth camera, microphone array, lidar and control module. The sound source positioning is performed through the microphone array, the binocular depth camera is visually recognized, and the three-dimensional one is combined with lidar and pitch rotation mechanism. Environment perception, using neural networks and object detection algorithms for location.

Benefits of technology

It has achieved accurate positioning of trapped people, improved rescue efficiency, shortened rescue time, and maintained stable movement and efficient inspection in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119084724A_ABST
    Figure CN119084724A_ABST
Patent Text Reader

Abstract

The invention relates to the field of indoor search and rescue, in particular to an indoor rescue vehicle based on audio-visual combination and a trapped person positioning method, and aims to solve the problems that trapped persons cannot be quickly and accurately positioned in an indoor rescue environment and rescue workers cannot efficiently check the scene in time, so that the rescue time is prolonged. According to the scheme, a pitching rotating mechanism is arranged on the upper portion of a rescue vehicle body, a sensor box is arranged on the pitching rotating mechanism, and a binocular depth camera, an information transmitting and receiving unit and a microphone array are arranged on the sensor box; the multiple Mecanum wheels are arranged at the bottom of the rescue vehicle body and driven by four motors installed on the rescue vehicle body, the control module and the laser radar are arranged on the rescue vehicle body, and the control module is used for completing signal interaction of the pitching rotating mechanism, the binocular depth camera, the information transmitting and receiving unit, the microphone array and the laser radar. According to the scheme, the audio-visual coordinates are corrected, so that the positioning is more accurate, and the rescue time is shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of indoor search and rescue, and in particular to an indoor rescue vehicle and a trapped person positioning method based on audio-visual combination. Background Art

[0002] With the deterioration of the ecological environment, the frequency of natural disasters is increasing, and the increasing size and complexity of buildings will make rescue after a disaster more difficult. Existing rescue equipment is not intelligent enough, large in size, and has poor mobility. It requires manual assistance, resulting in low rescue efficiency. After a disaster, it is necessary to quickly determine the location of the trapped people and carry out rescue. Therefore, indoor rescue vehicles are required to have the ability to perceive the three-dimensional environment and locate the target.

[0003] During the indoor rescue process, rescue workers mainly use rescue robots that use vision, ultrasonic sensors, and infrared methods to locate trapped people. However, due to the complex indoor environment caused by the collapse of objects, these methods make it difficult to quickly and accurately locate trapped people, resulting in the inability of rescue workers to quickly and accurately implement rescue.

[0004] In terms of hearing, there are various noises and reverberations in indoor rescue environments, and the sound may attenuate during propagation, which will reduce the accuracy of sound source positioning and take too long to locate. In terms of vision, firefighters currently use rescue robots that are mainly based on navigation control and combine color cameras with thermal imaging to locate trapped people in real time during most rescue processes. However, in an earthquake rescue environment, the collapse of indoor objects makes it impossible to visually see the trapped people and the indoor environment becomes complicated. In addition, a large amount of dust is generated in the air, causing the quality of the collected images to deteriorate, and the contrast and color information are missing, resulting in reduced visual recognition and positioning accuracy, and even the inability to identify trapped people.

[0005] In summary, the existing indoor rescue equipment and trapped person positioning methods are difficult to meet the needs of indoor rescue. In the indoor rescue environment, trapped persons cannot be quickly and accurately positioned, and rescue personnel cannot conduct efficient inspections on the scene in a timely manner, resulting in extended rescue time. Summary of the invention

[0006] In order to solve the problem that trapped persons cannot be quickly and accurately located in an indoor rescue environment and rescue personnel cannot conduct efficient on-site inspection in a timely manner, resulting in prolonged rescue time, the present invention provides an indoor rescue vehicle and a trapped person positioning method based on audio-visual combination.

[0007] To solve the technical problem, the solution of the present invention is:

[0008] An indoor rescue vehicle based on audio-visual combination, which includes a rescue vehicle body, a pitch rotation mechanism, a sensor box, a binocular depth camera, an information transmission unit, a microphone array, a control module, a laser radar and multiple Mecanum wheels;

[0009] The pitch rotation mechanism is arranged on the upper part of the rescue vehicle body, the sensor box is arranged on the pitch rotation mechanism, and the binocular depth camera, the information transmission unit and the microphone array are arranged on the sensor box;

[0010] Multiple Mecanum wheels are arranged at the bottom of the rescue vehicle body and are driven by four motors installed on the rescue vehicle body. A control module and a laser radar are arranged on the rescue vehicle body. The control module is used to complete the signal interaction among the pitch rotation mechanism, the binocular depth camera, the information transmission unit, the microphone array and the laser radar.

[0011] Further, the pitch rotation mechanism includes a base, a connecting rod 1, a connecting rod 2, a connecting rod 3 and a connecting rod 4;

[0012] The base is fixed on the top of the rescue vehicle body, two connecting rods four are symmetrically arranged and vertically fixed on the base, the upper part of each connecting rod four is hinged and driven to the lower part of a connecting rod three through a motor three, each end of the connecting rod two is hinged and driven to the side of a connecting rod three through a motor two, the connecting rod one is fixed on the connecting rod two, and the top end of the connecting rod one is hinged and driven to the sensor box through a motor one.

[0013] The method for locating trapped persons comprises the following steps:

[0014] Step 1: The microphone array is used to collect the distress voice signal sent by the trapped person, and the voice source signal is denoised by wavelet threshold. The threshold is adaptively selected using GAN, and the Hamming window is selected for windowing and framing processing; endpoint detection processing based on correlation method is adopted;

[0015] Step 2: Using the positioning algorithm based on SRP-PHAT and neural network, the response power spectrum of the voice source signal calculated by the SRP-PHAT algorithm is used as the input of the BP neural network. The output of the BP neural network is the coordinates of the sound source. The BP neural network locates the distress voice signal sent by the trapped person and obtains the auditory position coordinates of the trapped person.

[0016] Step 3: After obtaining the auditory position coordinates of the trapped person, the control module is used to adjust the pitch and rotation angles of the pitch rotation mechanism so that the visual position coordinates obtained by the binocular depth camera and the auditory position coordinates obtained by the microphone array are in the same coordinate system, and the laser radar is used for mapping and navigation. The rescue vehicle moves until the binocular depth camera can identify the trapped person;

[0017] Step 4: Determine whether to perform dust removal operation based on the image collected by the binocular depth camera; find out the approximate direction of the trapped person and determine a region of interest, called a ROI frame; then convert the RGB image format into the HSV format, and calculate the ratio u of the saturation s and the color value v in the HSV color model domain in the frame;

[0018]

[0019] By comparing u with the previously set value u0=3, if u>u0, the binocular depth camera (2) dust removal is performed to step five, otherwise, step six is ​​performed;

[0020] Step 5: The image is dehazed end-to-end using the AOD-NET dehazing network, and then processed using white balance, bilateral filtering, and gamma correction algorithms;

[0021] Step 6: Use the YOLOv5 target detection algorithm to identify and locate the trapped person based on the image, and obtain the visual position coordinates of the trapped person;

[0022] Step 7: Compare the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array to obtain the accurate position information of the trapped person;

[0023] Calculate the relative difference δ between the visual position coordinates and the auditory position coordinates:

[0024] δ=[(r1-r2-r3) / (r1-r2+r3) / 2]×100%

[0025] Among them, r1 is the voice source positioning distance; r2 is the moving distance of the rescue vehicle, that is, the distance the rescue vehicle moves from obtaining the auditory position coordinates to obtaining the visual position coordinates; r3 is the visual positioning distance, which compares the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array.

[0026] When the relative difference is less than 10%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; when the relative difference is greater than 10%, and when the confidence of visual target detection is greater than 80%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; when the confidence of visual target detection is less than 80%, the auditory position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person.

[0027] Furthermore, in the process of performing wavelet threshold denoising on the speech source signal in step 1, speech recognition is performed on the sound source signals emitted by the distress signals of "SOS" and "help", and such sound source signals are enhanced, and other detected sound source signals are weakened.

[0028] Furthermore, in step 2, the BP neural network is constructed by training data and based on the rules of mapping the corresponding power spectrum to the coordinates of the sound source position by the neural network learning.

[0029] Furthermore, the YOLOv5 target detection algorithm in step seven is:

[0030] First, by introducing the lightweight multi-scale mixed convolution LMSMC module, the network is made lightweight;

[0031] Then, a multi-pooling feature fusion module is introduced to replace the original SPP module to enhance the key feature extraction;

[0032] Finally, the convolutional attention module CBAM is introduced to improve the detection accuracy of the network.

[0033] Beneficial effects:

[0034] This solution is of great significance in achieving accurate positioning of trapped persons. It discloses an indoor rescue vehicle based on a combination of vision and audio, which has visual and auditory functions to search and rescue trapped persons. It is equipped with Mecanum wheels and can move on uneven roads. During the indoor rescue process, it can combine visual and auditory functions to locate trapped persons, and can also move stably under poor road conditions. Combined with the trapped person positioning method proposed in this solution, the visual and audio coordinates are corrected to make the positioning more accurate, and the trapped persons can be found as quickly as possible, which can assist rescue personnel in efficiently checking the rescue site and thus shorten the rescue time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the main structure of an indoor rescue vehicle based on audio-visual combination of the present application;

[0036] Figure 2 It is a schematic diagram of the lower part of an indoor rescue vehicle based on audio-visual combination of the present application;

[0037] Figure 3 This is a schematic diagram of the Mecanum wheel structure;

[0038] Figure 4 It is a flow chart of the trapped person positioning method of the present application;

[0039] Figure 5 is a flow chart of a method for determining location information of the present application;

[0040] Figure 6 It is a flow chart of the dust-free image target detection method of the present application;

[0041] Among them, 1 is a multi-sensor box, 2 is a binocular depth camera, 3 is an information transmission unit, 4 is a microphone array, 5 is motor one, 6 is connecting rod one, 7 is connecting rod two, 8 is motor two, 9 is connecting rod three, 10 is motor three, 11 is connecting rod four, 12 is a base, 13 is the rescue vehicle body, 14 is a Mecanum wheel, 15 is motor four, 16 is a control module, and 17 is a lidar. DETAILED DESCRIPTION

[0042] Specific implementation method 1: An indoor rescue vehicle based on audio-visual combination, which includes a rescue vehicle body 13, a pitch rotation mechanism, a sensor box 1, a binocular depth camera 2, an information transmission unit 3, a microphone array 4, a control module 16, a laser radar 17 and a plurality of Mecanum wheels 14;

[0043] The pitch rotation mechanism is arranged on the upper part of the rescue vehicle body 13, the sensor box 1 is arranged on the pitch rotation mechanism, and the binocular depth camera 2, the information transmission unit 3 and the microphone array 4 are arranged on the sensor box 1;

[0044] A plurality of Mecanum wheels 14 are arranged at the bottom of the rescue vehicle body 13 and are driven by a motor 15 installed on the rescue vehicle body 13. A control module 16 and a laser radar 17 are arranged on the rescue vehicle body 13. The control module is used to complete the signal interaction among the pitch rotation mechanism, the binocular depth camera 2, the information transmission unit 3, the microphone array 4 and the laser radar 17.

[0045] In this embodiment: The binocular depth camera model in this embodiment is Orbbec Gemini2XL. The microphone array adopts a planar distribution structure and includes 6 microphones, which can achieve 360-degree equivalent sound pickup and a wake-up resolution of 1 degree. The microphone array can be used to obtain the original and noise-reduced audio, the wake-up angle, and the main microphone number;

[0046] The control module includes an industrial computer, a driving circuit board and a wireless transmission unit. The industrial computer and the driving circuit board are connected via a serial port to issue instructions to control the activities of the indoor rescue vehicle. The indoor rescue vehicle can also be remotely controlled via the wireless transmission unit. The control module is configured to perform the following steps:

[0047] The lower part of the rescue vehicle chassis is equipped with an obstacle-crossing mechanism connected to four Mecanum wheels, which can overcome obstacles of small collapsed objects indoors, making the rescue vehicle more flexible.

[0048] Acquire the voice source information collected by the microphone array, and then use the positioning algorithm based on SRP-PHAT and neural network to obtain the auditory position coordinate data of the trapped person, and control the obstacle crossing mechanism and Mecanum wheel through the control module to reach the area where the visual module can identify the trapped person;

[0049] Obtain image information from the binocular depth camera, and then use the improved YOLOv5 target detection algorithm to obtain data information such as the visual recognition confidence and visual position coordinates of the trapped person;

[0050] The LiDAR sensor is installed above the chassis and connected to the control module. It cooperates with the microphone array to build maps and navigate. The four Mecanum wheels are directly driven by four motors, which can realize forward, reverse and obstacle avoidance in any direction of the indoor rescue vehicle.

[0051] Other implementations are the same as the first specific implementation.

[0052] Specific implementation method 2: An indoor rescue vehicle based on audio-visual combination, the pitch rotation mechanism includes a base 12, a connecting rod 1 6, a connecting rod 2 7, a connecting rod 3 9 and a connecting rod 4 11;

[0053] The base 12 is fixed on the top of the rescue vehicle body 13, and the two connecting rods 4 11 are symmetrically arranged and vertically fixed on the base 12. The upper part of each connecting rod 4 11 is hinged and driven to the lower part of a connecting rod 3 9 through a motor 3 10, and each end of the connecting rod 2 7 is hinged and driven to the side of a connecting rod 3 9 through a motor 2 8. The connecting rod 1 6 is fixed on the connecting rod 2 7, and the top end of the connecting rod 1 6 is hinged and driven to the sensor box 1 through a motor 1 5.

[0054] In this embodiment: the control module is used to adjust the movement of motor 1, motor 2, and motor 3 of the pitch rotation mechanism, and then the position of the sensor box is adjusted after the movement of connecting rod 1, connecting rod 2, connecting rod 3, and connecting rod 4.

[0055] Other implementations are the same as the first specific implementation.

[0056] Specific implementation method three: trapped person positioning method, step 1: collect the distress voice signal sent by the trapped person through the microphone array 4, perform wavelet threshold denoising on the voice source signal, use GAN to adaptively select the threshold, select the Hamming window for windowing and framing processing; use endpoint detection processing based on the correlation method;

[0057] Step 2: Using the positioning algorithm based on SRP-PHAT and neural network, the response power spectrum of the voice source signal calculated by the SRP-PHAT algorithm is used as the input of the BP neural network. The output of the BP neural network is the coordinates of the sound source. The BP neural network locates the distress voice signal sent by the trapped person and obtains the auditory position coordinates of the trapped person.

[0058] Step 3: After obtaining the auditory position coordinates of the trapped person, the control module 16 adjusts the pitch and rotation angles of the pitch rotation mechanism so that the visual position coordinates obtained by the binocular depth camera 2 and the auditory position coordinates obtained by the microphone array 4 are in the same coordinate system, and the laser radar 17 is used for mapping and navigation. The rescue vehicle body 13 moves until the binocular depth camera 2 can identify the trapped person;

[0059] Step 4: Determine whether to perform dust removal operation based on the image collected by the binocular depth camera 2; find out the approximate direction of the trapped person and determine a region of interest, called a ROI frame; then convert the RGB image format into the HSV format, and calculate the ratio u of the saturation s and the color value v in the HSV color model domain in the frame;

[0060]

[0061] By comparing u with the previously set value u0=3, if u>u0, the binocular depth camera 2 is used to remove dust and proceed to step 5, otherwise, proceed to step 6;

[0062] Step 5: The image is dehazed end-to-end using the AOD-NET dehazing network, and then processed using white balance, bilateral filtering, and gamma correction algorithms;

[0063] Step 6: Use the YOLOv5 target detection algorithm to identify and locate the trapped person based on the image, and obtain the visual position coordinates of the trapped person;

[0064] Step 7: Compare the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array to obtain the accurate position information of the trapped person;

[0065] Calculate the relative difference δ between the visual position coordinates and the auditory position coordinates:

[0066] δ=[(r1-r2-r3) / (r1-r2+r3) / 2]×100%

[0067] Among them, r1 is the voice source positioning distance; r2 is the moving distance of the rescue vehicle, that is, the distance the rescue vehicle moves from obtaining the auditory position coordinates to obtaining the visual position coordinates; r3 is the visual positioning distance, which compares the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array.

[0068] When the relative difference is less than 10%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; when the relative difference is greater than 10%, and when the confidence of visual target detection is greater than 80%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; when the confidence of visual target detection is less than 80%, the auditory position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person.

[0069] In this embodiment: the method is implemented in the following manner:

[0070] Step 1: The microphone array is used to collect the distress voice signal sent by the trapped person, and the voice source signal is denoised by wavelet threshold. The threshold is adaptively selected using the GAN generative adversarial network to effectively separate the noise and achieve better denoising performance; the Hamming window is selected for windowing and framing processing; and the endpoint detection processing based on the correlation method is adopted;

[0071] Step 2: Using the positioning algorithm based on SRP-PHAT and neural network, the response power spectrum of the voice source signal calculated by the SRP-PHAT algorithm is used as the input of the BP neural network, and the BP neural network outputs the sound source coordinates. The BP neural network locates the distress voice signal sent by the trapped person and obtains the auditory position coordinates of the trapped person;

[0072] Step 3: After obtaining the auditory position coordinates of the trapped person, the control module 16 adjusts the pitch and rotation angles of the pitch rotation mechanism so that the visual position coordinates obtained by the binocular depth camera 2 and the auditory position coordinates obtained by the microphone array 4 are in the same coordinate system, and the laser radar 17 is used for mapping and navigation. The rescue vehicle body 13 moves until the binocular depth camera 2 can identify the trapped person;

[0073] Step 4: Determine whether to perform dust removal operation based on the image collected by the binocular depth camera 2; find out the approximate direction of the trapped person and determine a region of interest, called a ROI frame; then convert the RGB image format into the HSV format, and calculate the ratio u of the saturation s and the color value v in the HSV color model domain in the frame;

[0074]

[0075] By comparing u with the previously set value u0=3, if u>u0, the binocular depth camera 2 is used to remove dust and proceed to step 5, otherwise, proceed to step 6;

[0076] Step 5: Use the AOD-NET dehazing network to achieve end-to-end dust removal on the image. Since the image dehazing algorithm based on the atmospheric scattering model will produce error accumulation in the parameter estimation process, resulting in dehazed image distortion, algorithms such as white balance, bilateral filtering and gamma correction are used to reduce the image distortion problem of the AOD-NET dehazing network.

[0077] Step 6: Use the YOLOv5 target detection algorithm to identify and locate the trapped person based on the image, and obtain the visual position coordinates of the trapped person;

[0078] Step 7: Compare the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array to obtain the accurate position information of the trapped person;

[0079] Calculate the relative difference δ between the visual position coordinates and the auditory position coordinates:

[0080] δ=[r1-r2-r3 / r1-r2+r3 / 2]×100%

[0081] Among them, r1 is the voice source positioning distance; r2 is the moving distance of the rescue vehicle, that is, the distance the rescue vehicle moves from obtaining the auditory position coordinates to obtaining the visual position coordinates; r3 is the visual positioning distance. Compare the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array.

[0082] When the relative difference is less than 10%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; when the relative difference is greater than 10%, and when the confidence of visual target detection is greater than 80%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; but when the confidence of visual target detection is less than 80%, the auditory position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person.

[0083] The relative difference formula is introduced to compare the position coordinates of the trapped person obtained by the auditory module and the visual module. When the relative difference is less than 10%, the result obtained by the visual module is used as the final position coordinate; when the relative difference is greater than 10%, and when the confidence of visual target detection is greater than 80%, the result obtained by the visual module is used as the final position coordinate; but when the confidence of visual target detection is less than 80%, the result obtained by the auditory module is used as the final position coordinate. The accurate position coordinates of the trapped person are obtained.

[0084] According to a large number of experimental verifications, when the relative difference between the results obtained by the sound source localization method in this paper and the improved YOLOv5 algorithm is less than 10%, the position coordinates of the trapped person obtained by the improved YOLOv5 algorithm are more accurate; when the relative difference is greater than 10%, and when the confidence of the target detection of the improved YOLOv5 algorithm is greater than 80%, the position coordinates of the trapped person obtained by the improved YOLOv5 algorithm in this paper are more accurate; when the relative difference is greater than 10%, and when the confidence of the target detection of the improved YOLOv5 algorithm in this paper is less than 80%, the position coordinates of the trapped person obtained by the sound source localization method in this paper are more accurate.

[0085] Relative difference = [speech source positioning distance - rescue vehicle moving distance - visual positioning distance / speech source positioning distance - rescue vehicle moving distance + visual positioning distance / 2] × 100%

[0086] The moving distance of the rescue vehicle is the distance the rescue vehicle moves from obtaining the auditory position coordinates to obtaining the visual position coordinates.

[0087] Other implementations are the same as the third specific implementation.

[0088] Specific implementation method four: The method for locating trapped persons, in step one, during the wavelet threshold denoising process on the voice source signal, voice recognition is performed on the sound source signals emitted by the distress signals of "SOS" and "help", and such sound source signals are enhanced, and other detected sound source signals are weakened.

[0089] In this implementation mode: in the process of performing wavelet threshold denoising on the speech source signal in step one, speech recognition is performed on the sound source signals emitted by the distress signals of "SOS" and "help", and such sound source signals are enhanced, and other detected sound source signals are weakened to reduce the interference of noise and reverberation to obtain a smooth speech signal, thereby improving the accuracy of sound source positioning.

[0090] Other implementations are the same as the third specific implementation.

[0091] Specific implementation method five: Trapped person positioning method, in step two, the BP neural network is constructed by training data and based on the rules of neural network learning corresponding power spectrum mapping to the coordinates of the sound source position.

[0092] In this embodiment: the trapped person is positioned using a trained BP neural network to obtain the auditory position coordinates of the trapped person.

[0093] Other implementations are the same as the third specific implementation.

[0094] Specific implementation method six: Trapped person positioning method, the YOLOv5 target detection algorithm in step seven is:

[0095] First, by introducing the lightweight multi-scale mixed convolution LMSMC module, the network is made lightweight;

[0096] Then, a multi-pooling feature fusion module is introduced to replace the original SPP module to enhance the key feature extraction;

[0097] Finally, the convolutional attention module CBAM is introduced to improve the detection accuracy of the network.

[0098] In this implementation: This solution improves the YOLOv5 target detection algorithm, which reduces the complexity of calculation to a certain extent, further improves the real-time performance of detection, and has a higher detection accuracy, meeting the requirements of real-time and accuracy in identifying and locating trapped persons in indoor rescue environments.

[0099] Other implementations are the same as the third specific implementation.

Claims

1. An indoor rescue vehicle based on audio-visual combination, characterized by: It includes a rescue vehicle body (13), a pitch rotation mechanism, a sensor box (1), a binocular depth camera (2), an information transmission unit (3), a microphone array (4), a control module (16), a laser radar (17) and a plurality of Mecanum wheels (14); The pitch rotation mechanism is arranged on the upper part of the rescue vehicle body (13), the sensor box (1) is arranged on the pitch rotation mechanism, and the binocular depth camera (2), the information transmission unit (3) and the microphone array (4) are arranged on the sensor box (1); A plurality of Mecanum wheels (14) are arranged at the bottom of a rescue vehicle body (13) and driven by a motor 4 (15) installed on the rescue vehicle body (13). A control module (16) and a laser radar (17) are arranged on the rescue vehicle body (13). The control module is used to complete the signal interaction among the pitch rotation mechanism, the binocular depth camera (2), the information transmission unit (3), the microphone array (4) and the laser radar (17).

2. The indoor rescue vehicle based on audio-visual combination according to claim 1, characterized in that: The pitch rotation mechanism comprises a base (12), a connecting rod 1 (6), a connecting rod 2 (7), a connecting rod 3 (9) and a connecting rod 4 (11); The base (12) is fixed on the top of the rescue vehicle body (13), and the two connecting rods (11) are symmetrically arranged and vertically fixed on the base (12). The upper part of each connecting rod (11) is hinged and driven with the lower part of a connecting rod (9) through a motor (10), and each end of the connecting rod (7) is hinged and driven with the side of a connecting rod (9) through a motor (8). The connecting rod (6) is fixed on the connecting rod (7), and the top end of the connecting rod (6) is hinged and driven with the sensor box (1) through a motor (5).

3. A method for locating trapped persons, implemented by the indoor rescue vehicle according to any one of claims 1 to 2, characterized in that: It includes the following steps: Step 1: The voice signal for help sent by the trapped person is collected through the microphone array (4), the voice source signal is denoised by wavelet threshold, the threshold is adaptively selected using GAN, and the Hamming window is selected for windowing and framing processing; Adopting endpoint detection processing based on correlation method; Step 2: Using the positioning algorithm based on SRP-PHAT and neural network, the response power spectrum of the voice source signal calculated by the SRP-PHAT algorithm is used as the input of the BP neural network. The output of the BP neural network is the coordinates of the sound source. The BP neural network locates the distress voice signal sent by the trapped person and obtains the auditory position coordinates of the trapped person. Step 3: After obtaining the auditory position coordinates of the trapped person, the pitch and rotation angles of the pitch rotation mechanism are adjusted through the control module (16) so that the visual position coordinates obtained by the binocular depth camera (2) and the auditory position coordinates obtained by the microphone array (4) are in the same coordinate system, and the laser radar (17) is used for mapping and navigation. The rescue vehicle body (13) moves until the binocular depth camera (2) can identify the trapped person; Step 4: Determine whether to perform dust removal operation based on the image collected by the binocular depth camera (2); find out the approximate direction of the trapped person and determine a region of interest, called a ROI frame; then convert the RGB image format into an HSV format, and calculate the ratio u of the saturation s and the color value v in the HSV color model domain in the frame; By comparing u with the previously set value u0=3, if u>u0, the binocular depth camera (2) dust removal is performed to step five, otherwise, step six is ​​performed; Step 5: The image is dehazed end-to-end using the AOD-NET dehazing network, and then processed using white balance, bilateral filtering, and gamma correction algorithms; Step 6: Use the YOLOv5 target detection algorithm to identify and locate the trapped person based on the image, and obtain the visual position coordinates of the trapped person; Step 7: Compare the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array to obtain the accurate position information of the trapped person; Calculate the relative difference δ between the visual position coordinates and the auditory position coordinates: δ=[(r1-r2-r3) / (r1-r2+r3) / 2]×100% Among them, r1 is the voice source positioning distance; r2 is the moving distance of the rescue vehicle, that is, the distance the rescue vehicle moves from obtaining the auditory position coordinates to obtaining the visual position coordinates; r3 is the visual positioning distance, which compares the visual position coordinates obtained by the binocular depth camera with the auditory position coordinates obtained by the microphone array. When the relative difference is less than 10%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; when the relative difference is greater than 10%, and when the confidence of the visual target detection is greater than 80%, the visual position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person; When the confidence of visual target detection is less than 80%, the auditory position coordinates are used as the final position coordinates to obtain the accurate position information of the trapped person.

4. The method for locating trapped persons according to claim 3, characterized in that: In the process of performing wavelet threshold denoising on the speech source signal in step 1, speech recognition is performed on the sound source signals emitted by the distress signals of "SOS" and "help", and such sound source signals are enhanced, and other detected sound source signals are weakened.

5. The method for locating trapped persons according to claim 3, characterized in that: The BP neural network in step 2 is constructed by training data and based on the rules of neural network learning corresponding power spectrum mapping to the coordinates of the sound source position.

6. The method for locating trapped persons according to claim 3, characterized in that: The YOLOv5 target detection algorithm in step 6 is: First, by introducing the lightweight multi-scale mixed convolution LMSMC module, the network is made lightweight; Then, a multi-pooling feature fusion module is introduced to replace the original SPP module to enhance the key feature extraction; Finally, the convolutional attention module CBAM is introduced to improve the detection accuracy of the network.